import opalstack

Posted On June 22, 2022

Editor’s note: Today we have a guest post from one of our developers and there’s tons of juicy technical content!

Introducing the open source Opalstack Python Library

Today, we are releasing an open-source Python library to make using our JSON API even simpler.

Almost everything that can be done through the Opalstack Dashboard can also be achieved in any programming language using our JSON-API. Over the past year, we have re-written and improved our API for simplicity and consistency; for instance, our MariaDB and PostgreSQL database APIs are now completely parallel, abstracting away significant differences in permissions management into a uniform structure. We’ve also improved performance by over 10x and written a comprehensive test suite to ensure that scripts you build today don’t break tomorrow.

We think Python is the perfect language for the types of automation tasks that benefit from an even-more streamlined API experience. Opalstack’s backend is written in Python and Go (as well as a little Rust and C). Internally, we have made extensive use of our own API for such things as certificate generation, deployment automation, and Dashboard integration. What we have found is that the easier it is to use the API, the smoother everything else becomes.

Run a script; get a site.

Let’s begin with a realistic example. Suppose you run a small investment services company for wealth-management advisors. Each advisor has clients who need a personal asset allocation that changes over time based on macroeconomic factors as well as their specific financial situation, time horizon, risk tolerance, insurance strategy, etc. What you want is to automatically generate a new website for each advisor when they sign up for your platform. Each of these generated sites needs its own subdomain, and you also want to isolate them so that one being hacked would not immediately compromise all of the others.

The first step is to build a template site with all of the functionality your customers need. The second step is to implement an upgrade process to import an older site into your latest platform version. Suppose you’ve just finished all of that. The next step is to automate the process of onboarding a new advisor by writing a script to generate a new subdomain and website for them. This is the part that turns your website into a platform.

In general, there are many reasons you may want to clone a website. Building a template site that gets cloned can form the basis of a platform like the one described above. It may also be useful to clone a website for the purpose of staging out changes or testing new hardware. We’ve built a full clone for WordPress using the Opalstack API library and provided it as an example. We chose WordPress first because it is very popular but also because it uses a database backend. The clone script demonstrates tools provided by the python library to assist in executing remote commands, transferring files, and importing and exporting database records. This example uses MariaDB, but tools exist for PostgreSQL as well. Therefore, this example can be used as a starting point for cloning other application types — Django, for instance.

Step 1: Install the opalstack python library:

pip3 install opalstack

Step 2: Get an API token:

Step 3: Download clone_wp_site.py example:

This example is designed to run from under the shell user containing the source application and clones it to a specified destination. It showcases a number of library features:

  • Retrieving webserver information using opalapi.servers.list_all() and filtering the results using filt_one().
  • Retrieving an existing shell user or creating a new one using opalapi.osusers.create_one().
  • Creating a domain, application, and site using opalapi.domains, opalapi.apps, and opalapi.sites, respectively.
  • Executing remote commands and copying files over SSH using opalstack.util.SshRunner.
  • Importing and exporting Maria databases using opalstack.util.MariaTool

Dynamic DNS

One of the simplest tasks which comes up in practice is the need to periodically update a DNS record to follow changes in a public IP address. This is a perfect task for a very small opalstack API script- so simple, in fact, that we can examine the next example directly:

The script creates a domain only if it does not already exist:

def get_or_create_domain(domain_name):
    domain = filt_one_or_none(
        opalapi.domains.list_all(), {'name': domain_name},
    )
    if not domain:
        log.info(f'creating domain {domain_name}')
        domain = opalapi.domains.create_one({'name': domain_name})
        assert domain
    return domain

The opalstack.util.filt_one_or_none() convenience function filters the output of opalapi.domains.list_all()), returning either None or exactly one (non-list) result.

Checking for an existing DNS record is straightforward also:

def dnsrecord_exists(domain, record_type, record_content):
    for dnsrecord in opalapi.dnsrecords.list_all():
        if ( dnsrecord['domain'] == domain['id'] and
             dnsrecord['type'] == record_type and
             dnsrecord['content'] == record_content ): return True
    return False

The opalstack library operates entirely with native python dictionaries, not classes representing each API model.

Finally, if the required DNS A record doesn’t exist, we create it:

opalapi.dnsrecords.create_one({
    'domain': domain['id'],
    'type': 'A',
    'content': ip_address,
})

The create_one() method creates a single item and blocks until it has been successfully created. (create() is the same, but takes a list of items. Analogous methods exist for update() and delete()).

Scripted Email

The opalstack API library includes email API endpoints as well. The final example is a command-line utility which creates a new email address and associated mailuser to deliver to:

A mailuser combines the concept of mailbox (for receiving mail) and SMTP user (for sending mail). The script receives an email address as a command-line parameter, creates the address domain, and then creates a mailuser with a specified prefix. This is to prevent conflicts, since mailusers must be globally unique. An existing domain and/or mailuser will be used if it already exists.

After creating the mailuser, it either creates a new email address or updates an existing one to configure it to deliver mail to the new mailuser:

address = filt_one_or_none(
    opalapi.addresses.list_all(), {'source': email_address},
)
if address:
    if mailuser['id'] not in address['destinations']:
        log.info(f'Updating address {email_address}')
        opalapi.addresses.update_one({
            'id': address['id'],
            'destinations': address['destinations'] + [mailuser['id']],
        })
else:
    log.info(f'Creating address {email_address}')
    opalapi.addresses.create_one({
        'source': email_address,
        'destinations': [mailuser['id']],
        'forwards': [],
    })

This is an example of updating an existing item. Updates work like creates except that the UUID of the object to update must be provided and any omitted fields will be left unmodified.

Conclusion

We hope that this has been a welcome and informative introduction to a new long-awaited feature for our developer community. A hidden world of internal and sometimes external API interfaces support a majority of nontrivial software systems in one form or another. Opalstack aims to solve a particular subset of these infrastructural needs and we welcome you to utilize these solutions where they improve simplicity. API access is fully included in your Opalstack subscription.