Work in Progress

Securing Spring Web Application with OAuth2

2025-01-16T00:00:00-05:00

It’s not uncommon to punt on account management when starting a new project and, instead, to add a database table to store accounts. The next step is to fill that table in with some sample data and move on to working on real features.

This might seem like an easy solution but inevitably this ends up becoming a core part of the application. It won’t be long before we find ourselves being asked to implement an administrative section where new accounts can be created and passwords changed. Eventually we’ll be tasked with coding up the ability to validate email address and reset passwords. The work around maintaining these accounts seems to never end!

Lucky for us there is an alternative: make someone else manage these accounts. 😉 At first this might seem like more work but I assure you, it is not. Vendors are lining up to implement all of these annoying features; things like changing passwords and validating email addresses or changing the account name, and from our point-of-view they are doing this for free. People interested in using your application might then register with their (Google, GitHub, Microsoft, etc.) account leaving you free and clear to focus on what your application actually does.

How Does It Work?

To keep things contained and manageable we’re going to use Keycloak instead of forcing you to register with Google or GitHub or whatever. We won’t spend a lot of time on managing Keycloak as that’s kind of out-of-scope but we will be giving you a place to start if you decide to use Keycloak for account management.

Next we’ll setup Spring’s OAuth2 Login support, forcing everyone that visits our application to be logged in. The way this works is that when someone hits our application, Spring will look to see if they are logged into our OAuth2 provider. If they are not, it will forward the browser to the provider so that it can handle logging the person in. Once logged in successfully the OAuth2 provider will then forward the browser back to our application, providing the account information to our application.

Lastly when people logout of our application we’ll clear their session data, we will also let our OAuth2 provider know so that it can invalidate that session for the account. This will keep things neat and tidy, once someone logs out they will need to log into the provider again in order to use our application.

Checkout the GitHub Project

To spare you the hassle of setting up Keycloak and a database server and creating a test account, I’ve done all of that work for you. All you need to do is checkout the project that goes with this article. It has a Docker Compose script and some seed data for Keycloak that will get you started.

https://github.com/cmiles74/minimal-spring-oauth2login

Pick a location on your machine and then go ahead and clone the project.

git clone https://github.com/cmiles74/minimal-spring-oauth2login

Aside from the demo project you will need the following installed and setup and working on your workstation.

Docker or Docker Desktop (if you’re on Linux, Docker, otherwise Docker Desktop)
Java JDK, version 21 or later
Git for source control management

That’s it! If you’re a Java developer (and you should be if you are interested in this article at all) then you probably have all of this stuff setup and working already. 😉

Bring Up the Supporting Services

Once you have the project checked out you’ll need to copy the default Docker environment variables file over. There’s no need to customize these but once you finish this article you’ll know where they are if you want to change them.

cp env-sample .env

With that done you can bring up the Docker stack.

docker compose up

This will download the PostgreSQL and Keycloak images, bring up their containers and then (in the case of Keycloak) load in some seed data. This seed data includes the “realm” and a demonstration account. The credentials for that demo account are…

Account Name: someone
Account Password: password

Keycloak will be running on port 8080, you can check it out at the following URL:

http://localhost:8080/

The credentials for the Keycloak administrator account on the “master” realm are…

Account Name: admin
Account Password: admin

Put the Pieces Together

Typically in a tutorial like this I’d make you create the files by hand and type in (or copy and paste) their contents. This one is a little different in that I needed to have you clone the repository to get the Docker stack and the sample data. Instead we’ll look over the key files and talk about their contents.

Add Spring Boot Dependencies

The first thing we need to do is add the Spring Boot dependencies that we know we’ll need to the project. If you look at the build.gradle file at the root of the project, you can see a pretty short and stock project. Down in the dependencies section you can see the four that we’ve added.

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter'
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'org.springframework.boot:spring-boot-starter-thymeleaf'
    implementation 'org.springframework.boot:spring-boot-starter-oauth2-client'
    developmentOnly 'org.springframework.boot:spring-boot-devtools'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
    testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}

We need spring-boot-starter because we’re using Spring Boot, the starter-web package let’s us build a web application. We then pull in start-thymeleaf so that we can display some simple pages. Lastly we add spring-boot-start-oauth2-client, this is the package that enables us to communicate with an OAuth2 provider.

Setup our Application Configuration

Our Spring application will need to know what to call itself, what port it should run on and how to communicate with our OAuth2 provider. All of this is configured in the application.yml file in the src/main/resources directory. The interesting bit here is the OAuth2 provider configuration.

security:
  oauth2:
    client:
      registration:
        keycloak-client:
          provider: keycloak-provider
          client-id: ${OAUTH2_CLIENT_ID}
          client-secret: ${OAUTH2_CLIENT_SECRET}
          authorization-grant-type: authorization_code
          scope: openid,profile
      provider:
        keycloak-provider:
          issuer-uri: ${OAUTH2_ISSUER_URL}

This configuration points our application at our local Keycloak server but all OAuth2 providers are configured in a similar manner. Take note of the stanza under registration and provider, we are naming our OAuth2 “registration” and provider (“keycloak-client” and “keycloak-provider”, respectively).

Our client identifier and secret are coming from your OAuth2 provider (and then stored in the .env file at the root of the project). The authorization grant type and scope will likely be the same for all providers. Lastly the issuer URI lets our application know where to collect information about the provider; this should be provided to you by your OAuth2 provider.

Configure Spring Security

Next we need to tell Spring Security how we’d like it to secure our application. We’ll have it require security for all pages (except the “goodbye” page) and let it know that we want to use OAuth2 to handle logging people in. This is in the SecurityConfig.java file in the src/main/java/com.nervestaple.demo directory.

@Configuration
@EnableWebSecurity
public class SecurityConfig {

    private ClientRegistrationRepository clientRegistrationRepository;

    @Autowired
    public void setClientRegistrationRepository(
        ClientRegistrationRepository clientRegistrationRepository) {
        this.clientRegistrationRepository = clientRegistrationRepository;
    }

    @Bean
    public SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
        http.authorizeHttpRequests(a ->
                        a.requestMatchers("/goodbye").permitAll()
                                .anyRequest().authenticated())
                .oauth2Login(Customizer.withDefaults())
                .logout(l ->
                        l.logoutSuccessHandler(oidcLogoutSuccessHandler())
                                .invalidateHttpSession(true)
                                .clearAuthentication(true));
        return http.build();
    }

    @Bean
    public OidcClientInitiatedLogoutSuccessHandler oidcLogoutSuccessHandler() {
        final OidcClientInitiatedLogoutSuccessHandler successHandler =
                new OidcClientInitiatedLogoutSuccessHandler(clientRegistrationRepository);
        successHandler.setPostLogoutRedirectUri("http://localhost:8085/goodbye");

        return successHandler;
    }
}

When someone logs into our application their information is stored in the client registration repository. We keep it simple here, if you want to learn more about how this works then checkout the Spring documentation.

Next we configure the security for our application with the SecurityFilterChain. This code is not so bad, you can see that we require authorization on all URLs except /goodbye, then we configure Spring’s OAuth2 Login (oauth2login) with default settings. Lastly we configure logout, in addition to clearing the session and authentication data we also invoke our OIDC logout success handler and redirect to the /goodbye URL.

With this configuration, two URLs are added to our application by Spring’s OAuth2 Login component.

http://localhost:8085/oauth2/authorization/keycloak-client, this endpoint checks to see if the browser is logged into our provider and forwards the browser to the provider for login if they are not
http://localhost:8085/oauth2/code/keycloak-client, this endpoint handles the response from our OAuth2 provider and forwards the browser back to our application

The last thing of interest is the oidcLogooutSuccessfulHandler, this bit of code is looking up the logged in account in our client registration repository and using that information to contact our OAuth2 provider to log them out of this particular session (the OIDC back-channel logout). If that’s successful then the browser is forwarded to our goodbye page.

Who is Logged In?

The rest of our skeleton web application is in the HomeController.java file in the src/main/java/com.nervestaple.demo directory. Here is were we setup our two endpoints (that serve HTML pages), /home and /goodbye. These are pretty dull and routine, the only interesting bit is in the home method.

@GetMapping("/")
    public String home(Model model) {
        var authentication = SecurityContextHolder.getContext().getAuthentication();
        var oidcUser = (OidcUser) authentication.getPrincipal();
        model.addAttribute("accountName", oidcUser.getPreferredUsername());
        return "index";
    }

We are requiring the browser be authenticated with our OAuth2 provider before they can view this page, we can get the security context and authentication data without fear. We then cast this to OidcUser (we only support OIDC authentication in this demo app, a more complex app would need to take more care) and then digout the name of the account that’s logged in so that we can say “hello”.

Building and Starting the Application

Building the application is easy with Gradle.

./gradlew build

This is a small application, it should build quickly and without issue. Now you can run the application.

java -jar build/libs//demo-0.0.1-SNAPSHOT.jar

Go ahead and visit the application!

http://localhost:8085/

You should be immediately directed to the login page of our OAuth2 provider (Keycloak in this case). Type in the credentials for the demo account…

Account Name: someone
Account Password: password

And you should now be directed to our page where the application says “hello” to you.

If you click on the logout link your session with the OAuth2 provider will be ended and you’ll be dropped at the goodbye page. If you go back to the application you will need to login once again.

Summing Up

We have very little code but we’ve done quite a lot! We have said goodbye to account management, instead letting our OAuth2 provider handle that for us. We have a functional and reasonably attractive log in and out process, which is also pretty secure to boot.

One thing to note is that we are not depending on the OAuth2 provider to handle things like idle log out. Once the browser comes back from the OAuth2 provider with a valid token we log them into our application, from then on our application takes responsibility for deciding when someone has been idle for long enough that they need to be logged out (this is a function of Spring Security and there are various knobs to adjust this).

In my opinion this is a reasonable way to do things; when your OAuth2 provider provides a token to your application that token is used both to identify the account as well as to access other services provided by the OAuth2 provider. That is, the expiration time on that token is meant to indicate when it’s no longer valid for use with the OAuth2 provider, it’s not saying anything about how your application functions. Likewise the refresh token provided is meant to be used if the expiration time has passed and more requests need to be made of the OAuth2 provider; in practice it’s best to get all communication with the provider out of the way immediately after the login.

I hope you find this tutorial useful! If you have any questions or comments or if you find any errors or mistakes please let me know ASAP. 🙂

Getting Started with Medley Interlisp

2024-07-08T00:00:00-04:00

I have long been interested in computing environments where the development tools are an integral part of the whole system. This started out ages and ages ago, when I started learning to write software in Java. When Sun released the first version of Java it was often compared to Smalltalk (in fact, IBM’s first Java IDE was written in Smalltalk). At that time the Smalltalk environments were more compelling and easier to use than Java and I was somewhat smitten.

I later discovered that the Lisp and Smalltalk environments of the 1980s were borrowing and swapping ideas and that they were very different from the “integrated development environment” (IDE) that I was using to write Java code. In these environments the developer was meant to write code that extended the system in some way. Applications you developed essentially became part of the environment and to a very real degree you built the tools that you needed to get your work done. Those tools could in turn be used by other developers and you could build on any functionality you ran across. This idea lives on in Emacs but I’m hard pressed to think of other tools that embrace pervasive extension.

Contrast this with computing today where we have so much more power at our fingertips but it’s much more difficult to put it to use (even if you are using Emacs 😉). Neither Windows or MacOS comes with development tools installed and both have spent a lot of time and energy making it difficult to customize the environment in any meaningful way, most people simply grow accustomed to whichever one they are using. For almost any task you need to install a separate application and each application (and it’s output) is deliberately silo-d off from every other. Even if you download their development tools, scaffolding up a new project is a time consuming endeavor that few have the motivation to pursue.

I can’t help but wonder if we will reach some inflection point and people will demand more flexibility out of their computing environments and the software they use. Why isn’t there a tool that let’s us manipulate our data, regardless of which application might have created it? Why is it so difficult to compose a sort of uber-document that includes text from a word processor, a couple of spreadsheets and interactive charts? With the pervasive networking environment we all live in, why so much frustration moving data from my laptop to my phone or, heaven forbid, someone else’s phone?

Taking a look at some of these old environments, like Medley, might give us some ideas of how things might be made to work more for us. Ways we can put the personal back in “personal computing”.

Getting Started with Medley Interlisp

In any case Medley Interlisp is one of these environments that you, the developer, are meant to mold to meet your needs. It comes with extensive documentation and it’s clearly designed to soak up all of your time and imagination. There is an ongoing project to get Medley running on current hardware and steady progress is being made every day, presently it’s at the point where you can run it on the major operating systems. When I started playing with it there wasn’t much in the way of tutorials, this articles tries to lay a little groundwork so that you can get started and, at the very least, get an idea of how it works.

You can run Medley in your browser or download and run Medley locally. I installed it locally but it doesn’t make much difference to this tutorial. If you do take the online route, you will need to create an account to save your data between sessions.

Once you have Medley installed you can invoke the “medley.sh” or “medley.bat” script to start the environment. A directory named “il” will be created in the root of your home directory, the state of the environment and files you create will be stored there by default.

Writing a Simple Program

Sad though it may be, it took me quite a while and plenty of Googling and YouTube watching in order to write a handful of functions and get them saved to disk! Lucky for you, reading this article is much simpler. To get you accustomed to the environment we’ll write a couple of simple functions to “greet” the person (you) and then save them to disk and load them back.

The Executive

Medley provides a window with a read-eval-print (REPL) loop that you may use to interact with the environment, it’s called the “executive”. You can type in some code and when you press the return key it’s executed by the environment and then you are prompted to type in more code.

The Medley environment supports multiple languages, in the picture above the “Exec” window is ready to accept Interlisp code; that’s the language we’re using in this article. If your “Exec” window is set for something else (i.e. Xerox Common Lisp, XCL) then you will want to create a new one.

To create a new Exec window, right-click on any empty space on your Medley desktop; a pop-up window with a list of applications you may run will appear. Slide down to the “EXEC” option and then over to the right to choose “Interlisp”. When you let go of your mouse button you will select the location and size of the Exec window by left-clicking and dragging until the window is the size you want.

If you need to move your window, right click on the window and then choose the “Move” option. Once you let go of the mouse button you can choose the new window location by left clicking wherever you want the upper-left corner of the window to be located. You can change the size of the window by left-clicking on the bottom-right corner of the window and simply dragging it around.

Simple Prompt Function

We need a function that will prompt people for input, we’ll then use this to ask people their name. Go ahead and type this code in at the “Exec” window.

(DEFINEQ (SIMPLE-PROMPT (prompt)
  (PROMPTFORWORD prompt NIL NIL T NIL NIL (CHARCODE (EOL)))))

Interlisp is case sensitive and most of the functions are in all caps as this was “the way” in the 1980s. To try to make the code easier to read I’ve written the variables out in lower case. Here we define a new function, SIMPLE-PROMPT, that accepts a string of text for the prompt and then passes it onto the PROMPTFORWORD function with a bunch of flags indicating that we want to collect input from the person at the console until they hit the return key.

Once you type in the code hit the return key, it will immediately be evaluated and ready for use. Go ahead and give it a try…

(SIMPLE-PROMPT "What is up? ")

Pretty neat!

Saving Your Work

Now that we have our function handy we want to save it to a file so that we can have it with us always. Interlisp has a function that will let you link functions in your environment with files on disk. Type the following at the REPL…

(FILES?)

The loose functions in your environment will be listed and you will be asked if you want to “say where the above go ?” which is Medley for choosing a save location; type “y” to agree. Next you will be asked where to save the SIMPLE-PROMPT function, type in “GREETING.LISP”. Medley will ask if you want to create a new file, press “y” to agree.

You might think the file is now present on disk but there’s one more step! Right now the function is linked to a file but that is all. The MAKEFILES function handles writing the functions out to the file. Type the following in the “Exec” window to save the function to disk.

(MAKEFILES 'GREETING.LISP)

Now the file is on disk, it should be in the “il” folder in your home directory.

Loading Your Work

When you need to load your functions back in, the LOAD function is there for you.

(LOAD 'GREETING.LISP)

Once loaded the functions will be immediately available.

Reading the Documentation

Medley provides a bunch of documentation and it’s really pretty good, this is how I found the PROMPTFORWORD function. To open a new documentation window, right click on some empty Medley desktop space and choose “DINFO”. Click somewhere to anchor the new DInfo window, two windows will appear! One will have the text of the documentation and the other a dynamic table of contents.

The scrolling of the windows is a little odd, you need to hover with the mouse pointer just a little past the left-hand margin of the DInfo window and the scrollbar will appear. You can click with the left mouse button to scroll down and the right mouse button to scroll up.

The IRM window has a dynamic table of contents, you can click on any of the items and the DInfo window will display the matching documentation. I found information on PROMPTFORWORD by clicking on “User Input/Output Packages” and then clicking on PROMPTFORWORD.

Write the Greet Function

We are almost done, the only thing left is to write our GREET function. This function will use our prompt function to get the person’s name and then it will say “hello” to them.

(DEFINEQ (GREET ()
  (LET ((person (SIMPLE-PROMPT "What is your name? ")))
    (PRINT (CONCAT "Hello, " person "!")))))

You may use the FILES? function to link the GREET function to your “GREETING.LISP” function and then MAKEFILES to dump it out to disk.

Editing a Function

Eventually you will need to edit an existing function. Medley has an editor you may use called “SEDIT” (Structure Editor). You can edit your GREET function like so…

ED (GREET)

You will need to select the location and size of the editor window by left clicking somewhere and dragging to set the window’s size. A new SEdit window will appear with your function’s code, ready for editing.

Like everything else it doesn’t work exactly as you might expect. The documentation linked above should be enough to get you started.

Good luck!

PICO-8 Animation and Collision Detection

2023-01-15T00:00:00-05:00

In my last article we spent some time getting our game shelled out, throwing a sprite for the player on the screen and moving it around a bit. What would really snazz that up, I think, would be a little animation of the player and ending the game if the player hits a hazard of some kind.

In this article we’ll add some animation to snazz up our player, introduce a hazard and then add some collision detection so that we can end the game if the player happens to smack into the hazard. Not a challenging game but it does have a beginning, middle and an end!

Animating the Player

What we would like is for the player to animate while it is moving around the screen. We’d like one set of animations while it’s moving along the “x” axis and another set while it moves along the “y” axis. To keep things simple we will only have two frames for each animation.

To make this work we’re going to setup the simplest possible animation system. We’ll have one variable that keeps track of each “tick” of the game (each cycle through our game loop); every time we complete a game loop we’ll increment the tick counter. When we reach 16 ticks we’ll start back at 0 and start all over again.

This simple system will give us 16 possible frames of animation, one per game loop cycle. That’s more than we need and it gives us a little room to grow if we decide to get clever.

Create Our Tick Counter

We’re going to build on the code from the previous article, either catch up by reading through it or just copy the relevant code out so that we’re all on the same page.

Let’s start by adding two variables to our init_actors function: one for the current tick and one for our maximum number of ticks. The code below will get this done.

function init_actors()
  atk=0 atx=15
  ply1={x=8,y=8,spr=0}
end

The atk variable stands for “animation tick” and starts off at zero. The atx variable stores our maximum number of ticks and we set it to fifteen, giving us sixteen ticks in total.

Next we need to increment and reset our tick counter. Let’s add a function to manage our animation counters.

function update_anim()
  atk+=1
  if atk>atx then
    atk=0
  end
end

Simple, right? We update our tick counter and if we hit our maximum number of ticks then we reset it back to zero.

Creating More Player Sprites

Next we need more sprites for our player, we want two to indicate movement horizontally and two more for vertical animation. Use the first four cells of sprites. You may also copy your sprite with the dotted square looking tool and paste it into the next cell to save time.

I’ve altered my sprites to change the direction the player is looking, I also change the fringe at the bottom of the ghost to give the impression of movement. The idea here is that the changes between the sprites don’t necessarily need to be dramatic to provide the illusion of motion.

Now we can associate our new sprites with our player. After thinking it over a bit, I decided to add two arrays to the player data: one with the horizontal frames and another with the vertical. Doing it this way, we could have different animations for the various characters in the game.

Update the init_actors() function to match the excerpt below.

function init_actors()
  atk=0 atx=15 
  ply1={x=8,y=8,spr=0,
    alr={0,1},aud={2,3}}
end

We’ve added two more variables to the ply1 table, alr and aud. Both of these contain an array with the index of the sprites that make up the animation. The alr variable (animation left/right) references the first two sprites (0 and 1). The aud (animation up/down) variable points to the third and fourth sprites (2 and 3).

We are now tracking the animation frames (ticks) and we have references to the sprites we want to use for the animation. The only bit remaining is swapping the sprites for the player as it moves around, we’ll do this by manipulating the ply1.spr variable that indicates which sprite to draw for the player.

Swapping the Player’s Sprite

Since we need to know which button the player is pressing to figure out which direction to move the player, we can save a little time by weaving our animation code into our button detection and movement code. We check if the left or right, or if the up or down buttons are being pressed, then we update the player’s position and alternate the player’s sprite.

function update_ply1()
  if btn(0) or btn(1) then
    if atk > 8 then
      ply1.spr=ply1.alr[1]
    else
      ply1.spr=ply1.alr[2]
    end
    
    if btn(0) then
      ply1.x -= 2
    end
    if btn(1) then
      ply1.x += 2
    end
  end
  
  if btn(2) or btn(3) then
    if atk > 8 then
      ply1.spr=ply1.aud[1]
    else
      ply1.spr=ply1.aud[2]
    end
    
  		if btn(2) then
      ply1.y -= 2
    end
    if btn(3) then
      ply1.y += 2
    end
  end
end

One game cycle goes by too quickly to really be visible, instead we change the sprite of the player every eight ticks. If we’re on a tick less then eight then we draw the first sprite otherwise we draw the second. You can also see in the code above that we use the ply.alr sprites if buttons 0 or 1 are pressed (left or right) and we use the ply.aud sprites if buttons 2 or 3 (up or down) are pressed.

That’s it! Run the game and give it a try, you should see your player 8-bit-ly animated as you move it around the screen. 😎

Hazards and Collisions

Now that we have our animated player moving around the screen, the next thing we need to be able to do is detect collisions between our player and the various hazards of the game. For that we need at least one hazard!

Adding a Hazard

Switch over to the sprite editor and add one more sprite, this one will be the thing that ends the game if the player were to touch it. It could be a hole or a bomb or an alligator, whatever you’d like. With that done, let’s add our hazard to the game. Let’s update the init_actors function to track our hazard and if the game is over (the player has collided with the hazard).

function init_actors()
  atk=0 atx=15
  ply1={x=8,y=8,spr=0,
    alr={0,1},aud={2,3}}
  hzd={x=64,y=64,spr=4}
  over=false
end

The hzd table has the “x” and “y” positions of our hazard along with the index of the sprite used to draw it on-screen. The over variable lets us keep track of the status of the game, we start out with it not being over.

Our hazard also needs to be drawn on screen, add this function right after draw_ply1.

function draw_hzd()
  spr(hzd.spr,hzd.x,hzd.y)
end

Simple right? We just draw the hazard on the screen. Update the _draw function to call this one as well.

function _draw()
  cls()
  draw_ply1()
  draw_hzd()
end

Now we’ll draw the hazard every loop of the game, just like the player.

Detecting Collision

Detecting a collision is really checking the distance between the player and the hazard, if they intersect then there’s our collision.

function collision(a,b)
  return (abs(a.x-b.x)+
    abs(a.y-b.y)) <= 8
end

The abs function returns the “absolute value”, the positive or negative information is discarded. For instance, 0 - 8 = -8 but with abs we just get back 8. For our purposes we want the distance between the two sprites, we don’t really care if it’s a positive or negative number.

Once we calculate the “x” distance and the “y” we add them up and check to see if it’s less than or equal to the width or height of our sprites, which is eight. If they are eight pixels apart or less then they’ve collided.

We can add our collision logic way down to the end of our update_ply1 function. Go ahead and add the code below directly before the end of the function.

  if collision(ply1,hzd) then
    over=true
  end

We check to see if the player has collided with our hazard and, if they have, we set the over flag to “true”. This indicates the game has ended.

Ending the Game

Now we need to let the player know that the game has ended! Let’s add a new function (right after draw_hzd) that displays “Game Over” on the screen.

function draw_over()
  if over then
    print("game over", 2, 2)
  end
end

As advertised, it checks to see if the game is over and, if it is, we display the game over text in the upper left of the screen.

The only thing we’re missing is and end to the action when the game ends. Right now, the player could keep moving around. Add the snippet of code below to the very beginning of the update_ply1 function.

if over then return end

If the game is over, we won’t update the player’s state. They’ll be frozen wherever they were when they collided with the hazard.

Whew! We didn’t add much code but we’ve definitely added a lot of functionality! Our player animates as they move and we introduced a hazard into our game. We added some collision detection, ending the game if the player collides with the hazard.

This might be enough to get you going on your own game. 😉

Getting Started with PICO-8

2023-01-07T00:00:00-05:00

I’ve been interested in fantasy consoles for a long time now but my interest has been limited to reading a blog article here and there and thinking something like “Man, that looks like fun!” After literal years of doing other things I’ve finally decided to put my own 8-bit arcade game together. I took a look at the field, tried the Pixel Vision 8 and then saw the project archived, regrouped and decided to go with the PICO-8!

At first I started out working with the Pixel Vision 8 project but I found some bugs that were frustrating me and while I was working on fixing those bugs, I found some more bugs. While I was thinking about fixing those bugs the project was archived, there’s currently no maintainer. Becoming the maintainer of one of these projects is out-of-scope, I’m more interested in getting a game coded up, so I decided to pony up the $15.00 and purchase a PICO-8 license. I recommend you do the same, its not that much money and it helps keep the project active.

It’s a pretty nice product and reasonably polished. It has handy tools for writing code, drawing sprites and tile maps, and editing music. While you could use external tools and pull in files, it’s nice to be able to do everything in one place. Plus it looks pretty dang awesome on my snazzy retro looking PC.

This post will cover getting the PICO-8 application installed, drawing your first sprite and moving that sprite around on screen. My plan is to add more articles as I get the game built up. I’m warning you now that this could take a while, I have a day job as well. 😉

Install the PICO-8 Application

Check all of the pockets of your coats and look under the sofa cushions until you’ve found $15 and then head over to the PICO-8 page. Once you fill in your e-mail address and hand over your dollars, you’ll end up with a license through the Humble website. You can then download the distribution for your operating system (or a ZIP of the distribution if you’re on Linux).

For those of you on Windows or MacOS, there’s not much to the installation; double-click the Windows installer or copy over the MacOS application from the disk image. Those of us on Linux will have to put the distribution somewhere and either start from the command line or create a “desktop” file to launch the application… Unless you are on Arch Linux, there is an AUR package for PICO-8 that’s pretty easy to get working.

With all of that work out of the way, you are ready to get started! Launch the PICO-8 application and you will be greeted with the welcome screen.

They’re definitely going for the old 8-bit home computer vibe, this reminds me of the Atari 800 or Commodore 64 where you’d get dropped at a BASIC interpreter. This is a bit better as the PICO-8 speaks Lua which is quite a bit more fun to work with.

What you are looking at right now is the interactive console, press the “escape” key on your keyboard to switch over to “editor” mode. It should look like the image below.

The icons in the upper-right hand corner represent the different editors: code, sprite, map, sound effects and music; this image is of the code editor. More information about all of these tools and how they work can be found in the PICO-8 manual.

Draw a Sprite

For now, switch over the the sprite tool (the second one from the left). For those of you not in the know, a sprite is a bitmap graphic that can easily be moved around on screen or stacked on top of or below other bitmaps. We will use this sprite to represent the player.

Each sprite on the PICO-8 is 8x8 pixels in size, these sprites are displayed in rows beneath the sprite editor area. There’s one sprite there already, it kind of looks like a white star. Click on that sprite to bring it into focus and display it in the editor area.

The sprites don’t have names but they each have a number: the first one (the one you’re editing) is 0, the one to the right of that would be 1 and the one to the right of that is 2 and so on.

There are several tools to make editing your sprite easier, feel free to explore them. Go ahead and edit the sprite until you have something you are somewhat satisfied with, after all, you can always go back and make changes later. This sprite will represent the player in your game.

Displaying Your Sprite

The next step is to write enough code to get your sprite to display when your game is run.

The Game Loop

The PICO-8 lets you write your game code with Lua. It’s pretty concise and, in my opinion, pretty easy to pick up from scratch; don’t worry if you don’t know much about Lua. You can read up on the language later, this tutorial will give you enough knowledge just-in-time style to get by.

Make sure you are in “editing” mode, press the “escape” key if you are not. Pick the code editor from the set of icons in the upper-right, it’s the first one and kind of looks like two parenthesis “()”. You will be welcomed by an entirely empty page of code, go ahead and bang in the code listed below.

function _init()

end

function _update()

end

function _draw()

end

This won’t actually do anything but it does provide a short game that does nothing! Press the “escape” key to switch back to console mode and type “run” (the run function will start your game) at the prompt. You should receive no errors and see nothing at all happen. When you grow tired of all the nothing, press the “escape” key to stop your game from running.

These three functions are common to every game, they are called by the PICO-8 to perform the following…

Setup your game
Update the current state of your game (i.e. move the player, the bad guys, etc.)
Draw the current state of the game on screen

While the setup function (“_init”) is only called once the other two are called over and over. When the “_update” function is called it can do things like see if any buttons are pressed and move the player to a new location. After that, the “_draw” function will be called to draw the current state of the game to the display. This is called “the game loop”, it’s a common pattern in a lot of different kinds of video games.

Draw the Player

While we could jam all of our code into those three functions, this style is frowned upon by sane people everywhere. Instead we’ll add some new functions to the top of our file and then call those functions where appropriate. First up is keeping track of the state of our player, add the following code to the top of your file.

function init_actors()
  ply1={x=8,y=8,spr=0}
end

This code creates a new variable named ply1 that keeps track of all the data about our player: their coordinates on screen (x and y) and the index of the sprite we’re using to represent them (spr). The data we’ve assigned to the player is called a “table”, it stores any number of keys and values and makes it easy to get at their values and update them.

Now we need to call our function to setuip our player along with the rest of the game. Edit your “_init” function to match the listing below.

function _init()
  init_actors()
end

Next up we want to draw the player. This will be pretty simple; we need to position the sprite in the player’s current location and then draw the sprite. Type in the code below right underneath your “initActors” function.

function draw_ply1()
  spr(ply1.spr,ply1.x,ply1.y)
end

The spr function is provided by the PICO-8’s sprite library, it handles all of the nitty-gritty of drawing the sprite to the display. It takes three arguments, conveniently all of the data it needs is available from our player data. We provide the name (numeric index) of our sprite, the coordinate on the “x” axis and then the coordinate of the player along the “y” axis. With that information in hand, our sprite will be drawn.

The last step is to update our “_draw” function to actually draw the player’s sprite. Edit your “_draw” function to look like the code listed below.

function _draw()
  draw_ply1()
end

Your entire file of code should look something like this…

Go ahead and save the code file and run your game. You should see your sprite on the screen. 😎

Move the Player

The last step is to let the player move around by pressing the arrow keys on their keyboard. There’s only a little bit of logic to handle, we’ll add it to a new function named “update_ply1”.

function update_ply1()
  if btn(0) then
    ply1.x -= 2
  end
  if btn(1) then
    ply1.x += 2
  end
  if btn(2) then
    ply1.y -= 2
  end
  if btn(3) then
    ply1.y += 2
  end
end

The PICO-8 has a btn function is doing most of the work here. This function accepts the index of button type (player 0 up, player 0 down, etc.) and returns true if that button is being pressed. We then check each button we’re interested in and then (if it’s being pressed) we update the state of the player to match.

To make this work, add a call to this function to our “_update” function.

function _update()
  update_ply1()
end

Try it out! You’ll notice it doesn’t work exactly as expected. The problem is that we aren’t clearing the display before we draw the sprite in the next location. The cls function will clear the screen, if we add that right before we draw the items on screen, we’ll be all set.

function _draw
  cls()
  draw_ply1
end

Woot! If you are interested in going the extra mile, how would we change the “update_ply1” code to let the player appear on one side of the screen when they reach the other, like in Pac Man? What would we do if we wanted the boundaries of the display to act as a wall? What might we change to make the player move faster or slower? There are many variations even for simple games. 🤯

Build Your Game

The last thing we’ll cover is building a “cartridge” that contains your game, this provides you one with one file that you can share with friends. You simply drag the cartridge file onto the PICO-8 (or executable) to load the game. With the PICO-8 we can also save the cartridge as an actual PNG image that you can view in an image viewer or web browser. It will have a screenshot as well as a little description of your game.

Switch to the code editor, we’re going to add a couple of lines to the top of your game’s code. We’ll add two comments; these won’t change how your game works but they will be used to describe your game and will be part of the cartridge. Two dashes in a row (“–”) start a comment, the comments are displayed at the bottom of your cartridge image. Add one comment with the name of your game and another with your name. Make sure these are the first two lines of code in the editor!

--move the player
--cmiles74

Next, switch back to the console and type “run” to run the game. Move the player around a little bit and then press the “control” key along with the “7” key (control-7). You should see the PICO-8 say “captured label image” down at the bottom of the screen. You’ve just taken the screenshot that will decorate your cartridge!

Now you can save your cartridge to disk. Type the following at the console to save your game.

save move-the-player.p8.png

At this point your game is built! You can switch to your operating system’s file browser so that you can navigate to your cartridge directory. This will vary based on your operating system.

Operating System	Cartridge Path
MacOS	~/Library/Application Support/pico-8/carts
Linux	~/.lexaloffle/pico-8/carts
Windows	%AppData%/Roaming/pico-8/carts

When you visit this directory you should see your cartridge listed. If you drag it over to your web browser, you’ll see what the image looks like.

You can load your cartridge by double-clicking it or dragging it onto your PICO-8 window. You can also email this file to friend or post it somewhere on the internet. Anyone with the PICO-8 application installed can run your game or play around with it, perhaps even making their own changes and sending you an updated cartridge.

Conclusion

It’s quite an achievement to get this much done so quickly, the tools provided by the PICO-8 make it easy to get straight to game building. Looking at your code, there’s not so much that it’s hard to read, hopefully it’s also pretty clear how things fit together.

Good luck on your next video game! If you are looking for inspiration, type “splore” at the PICO-8 console and check out some of the games people have written. Don’t forget to check back here for more articles, I plan to get a couple more written.

Assembling a PC in an old Macintosh SE Case

2022-12-19T00:00:00-05:00

Back before the pandemic I did all of my work from a laptop. I’d tow it back and forth between my home and work but, honestly, I mostly used it at work. The idea was that if there was a critical issue I’d have it handy and if I traveled to visit a customer it would be close at hand.

Things have certainly changed! Now I mostly work from my home office, my laptop spends most of it’s time plugged into a docking station powering a larger display, proper keyboard, etc. Many of the compromises the laptop represents no longer make as much sense. Why should I favor a small, light form factor when all it does is sit on my desk? Plus the harder it works, the warmer it gets and the more work it’s tiny internal fans spin, limiting it’s life span.

I decided to free myself from the shackles of the laptop, to build myself a real workstation that can easily handle my workload. From time to time I do travel, it would be nice to have something at least somewhat portable. If the day comes where I need to return to the office, it would be nice to be able take my PC with me. Maybe I’d take it home on weekends.

With these constraints in mind I started looking around at cases, looking for one that said “could maybe be moved” and “not entirely lacking in soul.” It turns out that there’s no shortage of featureless gray or black boxes but I wasn’t really excited about any of them. Then I thought back to the old Mac Classic II that I had in college, it was heavy but it had a handy duffel bag-ish thing I could lug it around in. Why not use that?

Decision made, I bought an old Macintosh SE case from a seller on eBay that was already bereft of guts and CRT. I measured the interior and was relieved to find that it would easily fit a mini-ATX board. All I had to do was 3D print some scaffolding to hold the pieces together and I’d be all set!

More easily said than done, but after about a year I finally have it finished up and working. Behold, my small but mighty six core AMD Ryzen 5 powered workstation!

I filled the top floppy drive bay with a USB hub. I’ve moved the power switch to the front, where the brightness knob once was is a nice, chrome push-button. It also has it’s own speaker, just like the old Mac SE.

An virtual machine running the MacOS of the era (although in color) is only a key-press away. 😯

The rear didn’t come out as nicely as I wanted but it could have been worse. 😛 I didn’t realize until far too late in the project that the logic board I purchased didn’t support USB-C video out, I had to run a cable to power the front LCD display. I also cut a little too low into the case for the ports and that looks pretty hacky. I may go back and print up a cover of some kind.

The scaffolding for the interior isn’t too complicated, I had to print the lowest layer in two parts in order to get the depth that I needed. The lowest layer holds the logic board and its components, the next layer supports the power supply. It was tricky getting a nice, snug fit on the front display that looked natural; I was saved by an adapter for the front of the case that worked out great.

After some searching around on the internet, I finally found a period travel bag that was in decent shape. After a little cleaning and repair, it’s good to go.

It’s been a month or so and this little guy is working out great. While its little LCD panel is on the small side, it’s not entirely useless. I’ve been parking my various chat windows (Teams, Slack, etc.) over there, keeping them out of my main work area. The low-profile Noctua CPU fan I picked out is silent most of the time and nearly silent when the machine is working hard, I haven’t seen any throttling.

So far it’s been a significant improvement over my laptop, I have no complaints. I’m also a little surprised at my schmalzy nostalgia for this case, I really enjoy seeing it perched on the edge of my desk.

Getting Started with Pixel Vision 8

2022-11-07T00:00:00-05:00

I’ve been interested in fantasy console for a long time now, but my interest has been limited to reading a blog article here and there and thinking something like “Man, that looks like fun!” After literal years of doing other things, I’ve finally decided to put my own 8-bit arcade game together. After taking a look at the field, I decided to go with the Pixel Vision 8! My plan is to try an get some of the people are work interested so it’s important that the console be free. 😉

It’s a pretty nice product and reasonably polished. It has handy tools for writing code, drawing sprites and tile maps, and editing music. It’s also pretty modular, each specialized function is managed by a “chip” (color, tile, sprite, etc.) that can be customized or even replaced with your own code. Plus, it looks pretty dang awesome on my snazzy retro looking PC.

This post will cover getting the Pixel Vision 8 application installed, drawing your first sprite and moving that sprite around on screen. If people are interested and if I have time, I may write another post that covers getting an actual game coded up.

Install the Pixel Vision 8 Application

Head over to the Pixel Vision 8 home page and click on the link for the “Latest Build”. This will take you to the page with the latest release on the project’s GitHub page. Scroll down to the “Assets” section and then click on the archive for your operating system. For instance, if you were on MacOs and the latest release was 1.0.18, you’d click on the link that reads “PixelVision8-v1.0.18-macOS.zip”.

Once you have the release downloaded, go ahead and unzip it. This is kind of on you, as every platform is a bit different. Good luck!

With the distribution unpacked, open up the directory and take a look at the contents. There’s a lot of stuff in there, you want to double-click the executable named “Pixel Vision 8”. The machine will boot up, load it’s operating system and present you with a welcome screen.

Pretty neat, eh? It kind of reminds me of the Amiga Workbench. Go ahead and click the “Yes” button to create a new project. You will be dropped at the “Workspace Explorer” where you can browse all of the files in your starter template project.

Draw a Sprite

First things first: we need a sprite to display on screen. For those of you not in the know, a sprite is a bitmap graphic that can easily be moved around on screen or stacked on top of or below other bitmaps. We will use this sprite to represent the player.

Sprites in Power Vision 8 are all 8x8 pixels in size, when your game starts up the file “sprites.png” is loaded and chopped up in to 8x8 images. Each of these images is then loaded into the memory of the sprite chip until it runs out of space. Each sprite is given an index number at load time (starting from 0) and you can use that index to fetch a particular sprite.

Double-click the folder labeled “Sprites” to open it up, then choose “New Sprite” from the “Workspace Explorer” menu (the menu is under an icon that looks like three colored slashes, “///”). The “New Sprite” dialog will appear, asking you to name the file; change the name to “sprites” (plural).

At this point we run into a minor bug in the Pixel Vision 8 Workspace Explorer: we need the “sprites” file in the root of our project, next to our “code” file. Unfortunately we can only create a new sprite file when we’re in the “Sprites” directory. Lucky for us this is easy to fix, simply drag the “sprites” file to the folder named “..”, the one with the green up-arrow icon. You will be asked to confirm the move, go ahead and say “yes”. You may now double-click the “..” folder to move up a level and you will see the files in your project.

Now that you have a file of sprite data, double-click the “sprites” file to open it. The sprite editor will open and you will be asked if you want to use the Pixel Vision 8 color palette, go ahead and say yes. You should now see one lone sprite in the upper-left hand corner of your canvas.

We want edit that little sprite, but it’s so small! In the lower right-hand corner of the editor is a large yellow button; click that button to zoom in and make the sprite as large as you can (if you click while holding down the shift key then everything will get smaller 😉). Now that it’s nice and big, edit that sprite until you think it looks like your player; you can select the draw color by clicking on it, the pencil and line tools may be used to draw your spite. The magenta background will be ignored, at game time it will be replaced with whatever is behind your sprite.

When you like the look of your sprite, choose “Save” from under the “///” menu to save your sprite and then choose “Quit” to close the sprite editor.

Displaying Your Sprite

The next step is to write enough code to get your sprite to display when your game is run. You should be back at the “Workspace Explorer”, looking at the contents of the “sprites” directory. Double-click the “code” file to edit your game’s code in the editor.

The Game Loop

The Pixel Vision 8 lets you choose between using Lua or C#, for this project we’ll be using Lua. It’s pretty concise and, in my opinion, easier to pick up from scratch than C#. You can read up on the language later, this tutorial will give you enough knowledge just-in-time style to get by.

You will see a bunch of code in the file already, this is the sample code that comes with each new workspace. In my opinion it’s not all that helpful; press “Control”+”A” to select all of the text and then hit the backspace or delete key to remove all of it. Next, type in the text below to code up the skeleton of your game.

function Init()

end

function Update(timeDelta)

end

function Draw()

end

This won’t actually do anything but it does provide a short game that presents an entirely black screen! Choose “Save” from under the “///” menu and then choose “Run Game” to try it out. Your game should load and you should see a screen of black. When you are done exploring this barren landscape, press the “Escape” key to quit your game and return to the editor.

These three functions are common to every game, they are called by the Pixel Vision 8 to perform the following…

Setup your game
Update the current state of your game (i.e. move the player)
Draw the current state of the game on screen

While the setup function (“Init”) is only called once the other two are called over and over. The “Update” function will be called, it can do things like see if any buttons are pressed and move the player to a new location. After that, the “Draw” function will be called to update what we see on the display. This is called “the game loop”, it’s a common pattern in a lot of different kinds of video games.

Draw the Player

While we could jam all of our code into those three functions, this style is frowned upon by sane people everywhere. Instead we’ll add some new functions to the top of our file and then call those functions where appropriate. First up is keep track of the state of our player, add the following code to the top of your file.

function InitActors()
  player = {posX = 8, posY = 8, sprite = 0}
end

This code creates a new variable named player that keeps track of all the data about our player: their coordinates on screen (posX and posY) and the index of the sprite we’re using to represent them (sprite). The data we’ve assigned to the player is called a “table”, it stores any number of keys and values and makes it easy to get at their values and update them.

Now we need to call our function to initialize our player when our game initializes. Edit your “Init” function to match the listing below.

function Init()
  InitActors()
end

function DrawPlayer()
  DrawSprite(player.sprite, player.posX, player.posY)
end

The DrawSprite function is provided by the Pixel Vision 8 (via the sprite chip), it handles all of the nitty-gritty of drawing the sprite to the display. It takes three arguments, conveniently all of the data it needs is available from our player data. We provide the sprite, the coordinate on the “x” axis and then the coordinate of the player along the “y” axis. With that information in hand, our sprite will be drawn.

The last step is to update our “Draw” function to actually draw the player’s sprite. Edit your “Draw” function to look like the code listed below.

function Draw()
  DrawPlayer()
end

Your entire file of code should look something like this…

Go ahead and save the code file and run your game. You should see your sprite on the screen. 😎

Move the Player

The last step is to let the player move around by pressing the arrow keys on their keyboard. There’s only a little bit of logic to handle, we’ll add it to a new function named “UpdatePlayer”.

function UpdatePlayer()
  if (Button(Buttons.Left, InputState.Down, 0)) then
    player.posX -= 2
  end
  if (Button(Buttons.Right, InputState.Down, 0)) then
    player.posX += 2
  end
  if (Button(Buttons.Down, InputState.Down, 0)) then
    player.posY += 2
  end
  if (Button(Buttons.Up, InputState.Down, 0)) then
    player.posY -= 2
  end
end

The Pixel Vision 8 Button function is doing most of the work here. This function accepts a button type (up, down, left, right, etc.), a state for the button, and a controller index. The state of the button is either “Down” (pressed) or “Up” (not pressed) and there are only two controllers (0 for player 1, 1 for player 2). When you call the “Button” function it takes this information and compares it to the actual state of the controller, if the controller is in that state then the function returns “true”.

When this function is called we check each button and, if it’s pressed, we update the current location of the player. Simple, right?

To make this work, add a call to this function to our “Update” function.

function Update(timeDelta)
  UpdatePlayer()
end

Try it out! You’ll notice it doesn’t work exactly as expected. The problem is that we aren’t clearing the display before we draw the sprite in the next location. The Clear function will clear the screen, if we add that right before we draw the items on screen, we’ll be all set.

function Draw()
  Clear()
  DrawPlayer()
end

Woot! If you are interested in going the extra mile, how would we change the “UpdatePlayer” code to let the player appear on one side of the screen when they reach the other, like in Pac Man? What would we do if we wanted the boundaries of the display to act as a wall? What might we change to make the player move faster or slower?

Build Your Game

The last thing we’ll cover is building a “disk” that contains your game, this provides you one with one file that you can share with friends. You simply drag the disk file onto the Pixel Vision 8 window (or executable) to run the game.

Go ahead and quit the code editor, bringing you back to the Workspace Explorer and viewing the contents of your game’s directory. First we want to change the name of your game and it’s description; double click on the “Info” file or edit your game’s information.

With this tool you may edit the name and description of your game, along with a couple other items. Go ahead and change the name and description, then choose “Save” from the “///” menu.

Next, click the “Build” button to the right of your game’s name. You will be asked if you really want to build the game’s disk, go ahead and click the “Build” button. Your game will be built and then you will be asked if you want to visit the new build directory, go ahead and click the “No” button and close the editor.

At this point your game is built! You can switch to your operating system’s file browser, there should be a “PixelVision8” directory present. When you open up that directory, you should see another named “Workspace”, this contains all of the files you were looking at with the Workspace Explorer. Open the “MyFirstGame” directory to see your game’s files, then navigate through the “Builds” and “Build” directories. You should now see the disk file for your game.

Drag the disk to the Pixel Vision 8 window. The disk with your game should be visible on the desktop of the Workspace Explorer and the files that make up your game should be visible. Now open the disk by double-clicking it and then pick “Run” from the “///” window: you will be able to run your game!

You can email this file to friend or post it somewhere on the internet. Anyone with the Pixel Vision 8 application installed can run your game or play around with it, perhaps even making their own changes.

Conclusion

It’s quite an achievement to get this much done so quickly, the tools provided by the Pixel Vision 8 make it easy to get straight to game building. Looking at your code, there’s not so much that it’s hard to read, hopefully it’s also pretty clear how things fit together.

Good luck on your next video game! If time allows, I’ll post another article that gets a little closer to a playable game.

Really Using Visual Studio Code Development Containers

2021-09-25T00:00:00-04:00

Here’s a hypothetical: you have to on-board another developer and, on top weathering their entirely reasonable questions about why this or that tool isn’t being used, you have to help them get their development environment setup. In addition to explaining that, yes, this other tool would be great and improve the project but time is kind of short right now and there are features that need to be developed you suddenly notice that they are not using Linux. All of your suggestions work, but not quite, and it is truly frustrating.

Development Containers

At this point, like all of us, you wonder “is there a better way?” There’s always a better way and the idea of using Docker to support your development environment is nothing new. The interesting bit is that Visual Studio Code is bringing some tooling to the table that makes this much easier.

A spec for Development containers or “devcontainers” to specify how to build your development container.
An extension called “Remote Containers” to make it easier to develop in these containers locally.
GitHub Codespace will run your development container in the cloud and connect remotely from Code or through their website.

Before we go further, let me clarify: a development container isn’t just one container. You can also provide a Docker compose file where you can describe all of the other containers your need to work on your project (a database server, web server, etc.)

Put this all together and every new developer on your project is literally one click away from being productive.

Fastest: They can click the link on GitHub and use the web-based version of Code and get to work within minutes.
Fast: They can click one more button to spin up the required containers on GitHub and then connect from their own, locally installed and customized copy of Code.
Somewhat Quickly: They can clone the project and open it in their local copy of Code, letting Code provision the containers on their local machine
Not quick at all: For the truly picky developers, they can do whatever they like on their own time, we’ve already provided three working options!

In my opinion, they real pain-point we’re avoiding here is the morass of confusion and mediocre performance that is Docker Desktop. I could detail my complaints about the product but the fact is that if you’re reading this article, we’re all on the same page already.

Getting It Setup

If you’re on the fence about Visual Studio Code, then just put your concerns aside and trust me; it’s worth it. It’s free, it’s popular, it’s easy to install and there are extensions available for nearly every feature it lacks. It may not be every developer’s preference but it’s easily good enough to get the job done.

Next install (if you haven’t already)…

Visual Studio Code
Git
Docker (if you are on Linux) or Docker Desktop (if you’re on MacOS or Windows)

After that, the first step is to get the development container setup and working. You can start from scratch by creating a directory for your sample project (maybe called “new-project”, avoid spaces in the directory name) and then a directory inside of that called .devcontainer with a file called devcontainer.json inside of that. The simplest file will contain only a pointer to a Docker image. Using Code, create and save the following devcontainer.json file.

{
  "image": "mcr.microsoft.com/vscode/devcontainers/php:8-bullseye"
}

That will spin up a development container based on Microsoft’s PHP image, which in turn is built on Debian Bullseye. To try it out, choose “Command Palette…” from under the “View” menu and start typing “Developer: Reload Window” to choose that command.

Code will close and re-open the sample project and then it will prompt you to install the recommended extensions. Click on the “Install” button.”

Now that the extension is installed, Code will suggest that you re-open the project one more time but, this time, open it in a container.

This time Code will open up the project and then spin up a new Docker container using the image we specified in the .devcontainer.json file. It will then install it’s own set of tools inside that container to make it easier to edit files, run code, etc.

When it’s all done you’ll be back at the a regular Code window, with your project files on the left-hand side. But if you select “New Terminal…” from under the “Terminal” window, you’ll see that the terminal has opened a session inside of the container!

Customize the Development Container

This is the simplest scenario where we just plop a development container definition into our project and Code spins it up. But what if we needed to add some of our own tools? Perhaps our team relies heavily on Flyway for database migrations and HTTPie for testing our RESTful endpoints. This can easily be done by writing our own Docker image definition. First remove the “image” attribute from your devcontainer.json file and replace it with a “dockerFile” attribute that points to our soon-to-be-created Dockerfile.

{
    "dockerFile": "Dockerfile"
}

Next, create a new Dockerfile next to your devcontainer.json file and add the following recipe to it.

FROM mcr.microsoft.com/vscode/devcontainers/php:8-bullseye

# update apt
RUN apt update

# install httpie
RUN apt install -y httpie

# install flyway
WORKDIR /usr/local
RUN wget -qO- https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/8.0.0-beta1/flyway-commandline-8.0.0-beta1-linux-x64.tar.gz \
| tar xvz && sudo ln -s `pwd`/flyway-8.0.0-beta1/flyway /usr/local/bin

Instead of using the PHP image directly we use it as a starting point for our own container image. We run apt update to pull down the latest list of packages for the distribution and then we use apt to install HTTPie. With that done, we follow the instructions from the Flyway website to get that tool installed as well.

Save the file and then reload the window (by choosing the option from the command palette). Code will close and then open the project and prompt you to open the project yet again in the container, go ahead and do so. While it’s doing that it will notice that the development container definition has changed and ask you if you’d like to rebuild it, do that as well.

Code will delete the Docker image, rebuild the image using our Docker file and then drop us at the project window. You can open up a new terminal panel to verify that our will truly has been done and that our tools are now available.

At this point you should start to get excited, thoughts of time saved and frustrations avoided racing through your mind. But wait, there’s more!

Add More Containers to the Development Environment

If we have anything at all in common, your next though is something along the lines of “how do I get more containers in there, for things like a database or a web server?” In the same way we added a Dockerfile we can also add a Docker Compose definition. When Code starts up the project it will bring up all of the containers in the compose definition and then connect to the development container. Let’s add a database server to our devcontainer.

Open up your devcontainer.js file, remove the dockerFile attribute and replace it with a dockerComposeFile entry.

{
  "dockerComposeFile": "docker-compose.yml",
  "service": "development",
  "workspaceFolder": "/workspace"
}

We point our devcontainer definition at our Docker Compose file, with the service attribute we let Code know which container we’d like it to connect to when that stack for the project is brought up. Lastly, we add a workspaceFolder attribute and provide the complete path where we want our project files mounted.

We also need to create that compose file. Create a new file called docker-compose.yml right next to our devcontainer definition and add the following.

services:
  development:
    build: 
      context: .
      dockerfile: Dockerfile
    volumes:
      - ..:/workspace
    stdin_open: true

In the stanza above, we describe the container that we’ll be using for our development environment. We point it at our existing Dockerfile and we map in the volume that will hold our project files (the directory above the one holding our devcontainer definition, mounted at “/workspace”). The context attribute tells compose to look for files in the current directory, stdin_open indicates that we’ll be connecting with an interactive session.

That takes care of our development container, now we’ll add a database server. Since we’re using PHP for this project it’s clear that the database needs to be MySQL. Before we add the database service to our compose file, let’s setup the credentials we’d like to use. Docker Compose will read a file named .env when it starts assembling the stack and it will make the variables in that file available to our script. Create a new file named .env next to your docker-compose.yml file and add the text below.

DATABASE_HOST=mysql_db
DATABASE_USER=example_account
DATABASE_PASSWORD=applesauce
DATABASE_DB=example

The database credentials will now be in environment variables that we can use in our compose file. Add the following service you your docker-compose.yml file to get our database server setup.

  mysql_db:
    hostname: mysql_db
    image: mysql/mysql-server:latest
    environment:
      - MYSQL_ROOT_PASSWORD=${DATABASE_PASSWORD}
      - MYSQL_DATABASE=${DATABASE_DB}
      - MYSQL_USER=${DATABASE_USER}
      - MYSQL_PASSWORD=${DATABASE_PASSWORD}

Before we spin this up, it would be nice to have an convenient way to interact with our database server from inside Code…

Add Visual Studio Code Extensions

In fact, it would also handy to have Code extensions that we’ve agreed to use in our project installed along with everything else. Let’s add the PHP extension pack and then a MySQL client. Open up the your devcontainer.json file and add the extensons attribute listed below.

{
    "dockerComposeFile": "docker-compose.yml",
    "service": "development",
    "workspaceFolder": "/workspace",
    "extensions": [
        "felixfbecker.php-pack",
        "cweijan.vscode-mysql-client2"
    ]
}

Open the command palette and ask Code to reload the folder. Code will prompt you to rebuild the devcontainer as the configuration files have changed; agree and let it bring up the new environment. When it’s finished you will be back at your Code window, the new extensions will be listed in the “Extensions” panel under “Dev Container - Installed”.

You can click on the icon for our database client and type in the connection information (remember the “hostname” attribute in our compose file?) to connect to our new MySQL database.

Now we’re starting to get somewhere! We have a container we can use for development, with all of our tools and Code extensions installed and another container running our database server. From here we could add a third container to run our web server and serve our application. We could even add migrations to get our application in a usable state, ready for developers to start working on new features.

This article is already getting pretty lengthy, we won’t detail getting migrations or a web server working here. You can take a stab at it on your own or you can checkout the full example project on GitHub.

Complete example project with web server and migrations

GitHub Codespace

The last thing I wanted to mention is that we have everything we need to host our development environment on GitHub Codespace. If we publish this on GitHub, our devcontainer definition will get picked up and (if it’s enabled for your account) the “Codespace” tab with a “New codespace” button will appear for anyone who stumbles upon the project. Anyone who would like to contribute can simply click the button and start work in seconds. Codespace will use the their GitHub credentials to handle pushing, pulling and committing code.

Summary

I encourage you to checkout the example project on GitHub, you should try it out in Codespace as well as launching the stack in Codespace and then connecting with your own, local version of Visual Studio Code. In my opinion this latter option is pretty compelling: you can customize Code the way you like it (including extensions that you prefer) and then connect to a set of running containers hosted at GitHub.

This is especially compelling if you are on a platform that only provides Docker through Docker Desktop (MacOS or Windows). While Docker Desktop is fine for small projects, anything of significant size will bring Docker Desktop to it’s knees. The real pain point here is sharing files from the host machine to the virtual machine that’s running Linux (and Docker) and then sharing those files once more with the actual containers running in Docker.

I hope this article has given you an idea of how Code development containers work and how they might fit into your own workflow. In my opinion, it’s an exciting feature and really gets us to a place where using Docker to containerize our development environment starts to become accessible to everyone, regardless of their level of experience with Docker and the related tooling. While Codespace might feel a bit gimmicky, it does allow anyone who may want to contribute to your project a quick and easy way to get started and to submit that pull request.

Getting Started with Clojure, Even on Windows

2021-04-29T00:00:00-04:00

When you find yourself looking to flex your language learning skills, functional languages often end up on your short list. After reading this past year’s “State of the Clojure Community” survey results, I couldn’t help but notice how few developers who run Windows are choosing to learn Clojure. For sure, it could be the shining allure of F#. But maybe it’s also that many tutorials focus on Linux or MacOS and often skip over the installation of the Java development kit. In any case, this article is going to cover getting up-and-running with Clojure, specifically when using Windows and we won’t be using the Windows subsystem for Linux (WSL).

In my opinion, Clojure has a lot to offer anyone working on software development regardless of the operating system we have on our primary machine. :wink:

Aside from being a functional language, Clojure has some other interesting features: there is a very real stress on immutability and that makes some other things, like concurrent or parallel code, easier to reason about. The language continues to evolve and improve and the community is reasonably friendly and accessible.

Tools

This article assumes that you have little or no experience with Clojure and that you haven’t really done much in the way of setup ahead of time. To that end we’ve chosen some tools for you.

Visual Studio Code for editing source code
Calva will be the VS Code plugin we use to make working with Clojure straightforward
Clojure runs on the Java virtual machine, we’ll get our JVM from AdoptOpenJDK
There are a couple different tools for managing Clojure projects, we’ll use Leiningen

All of these are quick and painless to install, even under Windows.

Install Your Tools

We’re going to start at the bottom of our list and work our way up, the first thing we will install is…

Java

Open up your preferred web browser and navigate to the AdoptOpenJDK website. Whatever they reccomend is fine, leave the radio buttons as-is and click on the “Latest Release” button to download the Java Development Kit installer. Double-click the installer to run, accept the license agreement, and install the software.

Leiningen

Clojure projects are typically managed by a tool, we’ll be using Leiningen. It will be responsible for downloading and managing libraries that we use in our project and building our code during development and deployment.

Where Do You Keep Your Scripts?

Leiningen is distributed as a batch file that you place in your path. This is pretty common and if you have a location for storing files like this, you can skip this part. If you don’t have a location, read on!

I recommend that you store your loose executables and scripts in a directory named “bin” in your home directory. We will now create that folder and then add it to your path, making these scripts available to you from wherever you happen to be on your machine.

Open up a File Explorer and then select the directory with your name from the right-hand side to view the contents of your “home” directory (“C:\Users\YOUR_NAME”). Create a new directory for your loose scripts and call it “bin” (C:\Users\YOUR_NAME\bin”), you can double-click it to view it’s contents and then click on the path at the top of the window and copy it to the clipboard.

Next, right click on the icon for “This PC” and choose the “Properties” option; the details for your computer will appear. Click on the “Advanced System Settings” item in the right-hand menu, then click on the “Environment Variables” button in the lower right-hand corner. The “Environment Variables” window will appear with the entries specific to your account at the top. Highlight the entry with the value “Path” in the “Variable” column and then press the “Edit…” button. You will be present with a list of all the directories added to your path, click the “New” button to add a new entry and then paste in (or type) the path to your new “bin” directory (“C:\Users\YOUR_NAME\bin”). Press the “OK” button to make the change permanent and then again to dismiss the “Environment Variables” window and then again to close the “System Properties” window.

Install Leiningen

Installing Leiningen is straightforward: we download the script and then place it in the same directory where we keep all of our loose tools and scripts (i.e. “C:\Users\YOUR_NAME\bin” or any location on your executable path). Open your web browser and navigate to the Leiningen website and scroll down to the “Install” section, you will see the instructions and links to the two scripts (one for Linux and one for Windows). Right click on the script for Windows (“lein.bat”) and choose the “Save Link As…” item to download a copy to your machine. The last step is to place the file in your “bin” directory (“C:\Users\YOUR_NAME\bin”).

Now the Leiningen is available, we can tell it to download its own dependencies so that it’s ready to run. Open up a Powershell window and then run the following:

PS > lein self-install

The self-install should go pretty quickly, typically the only output you’ll see is…

Downloading Leiningen now...

Just to test it out, you can ask Leiningen what version it is.

PS> lein -version
Leiningen 2.9.6 on Java 11.0.11 OpenJDK 64-Bit Client VM

Visual Studio Code

You probably have Visual Studio Code installed and running already, but for the sake of completion we’ll provide a brief overview of that here. If you’re all set with Code, you can definitely skip this step.

When you visit the Visual Studio Code download page, you can click on the big blue “Windows” download box. That will get you the “User Installer” for your machine. This version will install everything in your current account profile and, in general, is much easier to keep up-to-date.

Once downloaded, you can double-click to start the installation. After you accept the license and choose the install location, you will be presented with a couple other options. All things being equal, I recommend that you check the boxes to add an “Open with Code” action to Windows Explorer files and directories.

Install the Calva Clojure Plugin

Calva is a plugin for Visual Studio Code that works with Leiningen (and other tools) to provide an integrated development environment where you can work on your Clojure code and get real work done. Since we already have our Java development environment and Leiningen installed, this will be very easy!

Open up Visual Studio Code and then click on the “View” menu and choose the “Extensions” item. The “Extensions” panel will load in the left hand side of the editor window and a list of popular extensions will be displayed. Click in the search box at the top of the panel and type in “Clojure”. You will see that there are a variety of plugins for Code that support Clojure, “Calva” should be listed towards the top. Go ahead and click on “Calva” to select it, information about the plugin should appear in the main body of the editor window. Click on the “Install” button at the bottom of the Calva header panel to install it.

Create Your Project

Now that we have all of the tools installed, we are ready to create our project! Close the editor panel for Calva and then click on the “View” menu and choose the “Explorer” option. Next, click on the “File” menu and choose “Open Folder…” A new open dialog will appear, navigate to where you would like to store your code and then click the “New Folder” button towards the top of the window to create a new folder, call it something like “tutorial”. Next, click on the new folder to select it and click on the “Select Folder” button at the bottom of the window.

Create the Project File

Every project needs a project file where we can store information about our project, like it’s name, version and any libraries that it depends on. Click on the “File” menu and choose “New File” to create a new file. Before we start adding content to the file, choose “Save As…” from under the “File” menu and name your file “project.clj”, you want to save it right into the root of your project directory.

We’ll fill this file in with our project definition. You can model on the very simple one listed below (or copy it directly).

(defproject tutorial "0.1"
  :dependencies [[org.clojure/clojure "1.10.3"]]
  :main tutorial)

Here we set the name of our project, it’s version and a description. Next we add the version of Clojure we’re working with as a dependency (1.10.3 is the current version at the time of this writing). On the last line we point out which file should be the entry point to the project. You can name it anything, in this case we called it “tutorial” meaning that Leiningen should look for a file called “tutorial.clj” in our source code directory.

Add a Source File

Now we can add a file of source code. Leiningen will look for a source code in the “src” directory; in the “Explorer” panel on the left-hand side of your Code window, you will see some icons appear at the top when the mouse enters the panel or it gains focus. Click on the icon that looks like a folder to create a new directory and call it “src”. Click on that new directory and then click on the icon that looks like a file to create a new file inside that “src” directory and call it “tutorial.clj”. Add the text below to your “tutorial.clj” file and then save it.

(ns tutorial)

(defn -main
  [& args]
  (println "Hello!"))

The “-main” function represents the entry point to our application, if we were to compile our project and run it on it’s own, the “-main” function is called with any parameters we pass in on the command line.

Start An Interactive Session (REPL)

Select the “Command Palette” item from under the “View” menu, the command window will appear at the center top of your Code window, it will have a text area at the top where you can type in text and a list of matching commands in the body of the window. Type in “jack” and you will see the that the list narrows by a lot, you will see the “Calva: Start a Project REPL and Connect (aka Jack-in)” item in that list, to the right of it will be the command shortcut to invoke the task (Control-Alt-C and then Control-Alt-J). Note the shortcut as it’s easier to type then navigating the command palette and then select the “Calva: Start a Project REPL…” item.

Calva will now ask you what kind of REPL session you’d like to start. Go ahead and pick “Leiningen” (the first item) to start a new REPL with the Leingingen tool. The panel at the bottom of your window will show you details as Calva sets up and connects to your REPL and then a new editor pane will appear with the contents of your session (typically called “output.calva-repl”).

Go ahead and click into the REPL pane to make it active. You’ll see some introductory text from Calva and then the prompt (clj:tutorial:>). Type the following after the prompt and press return.

(-main)

This will call the main function and you should see output similar to what is pictured below.

; Jack-in done.
clj-tutorial-> 
(-main)
Hello!
nil
clj-tutorial->

Congratulations! You have installed all of the tools and started a new Clojure project. :trophy:

Working with Calva

There’s a good amount of documentation on the Calva website, you should definitely check it out. There is a “try first” tutorial that goes over compiling your whole file or just the form you are working on. From there you can explore the rest of the documentation to get productive in no time.

Build a Self Contained File

Now that we have some code that knows how to say “hi”, the last thing we’ll do is build one Java archive (JAR) file that contains our application. We can then distribute that archive to anyone else who would like to run our code.

First we need to compile a class with our entry point so that a regular Java runtime knows how to start it up. Open up the “src\tutorial.clj” file and add (:gen-class) to the namespace declaration at the top, like so:

(ns tutorial
  (:gen-class))

(defn -main
  [& args]
  (println "Hello!"))

Next we’ll ask Leiningen to compile one archive for our project. If we had included any libraries in our project, Leiningen would add those to the archive as well. Click on the “View” menu and choose the “Terminal” item, a new terminal pane will open at the bottom of your code window. Type in the following and press return to build a self contained JAR file for your project.

C:\> lein uberjar

Leiningen will display some output and build your archive. You may run it like so…

C:\> java -jar target\tutorial-0.1-standalone.jar

You should see the output with the text “Hello!” At this point you should know enough to be dangerous. Good luck! :smirk:

Provision Pragmatically and Programmatically with CloudFormation, 3

2019-04-22T00:00:00-04:00

This is the third and final part in a three part series where we build up a CloudFormation tempate for what I like to think of as a pretty typical environment: one virtual private cloud broken into two subnets and including two instances. So far we have covered…

Part 1: we setup of the VPC and subnets
Part 2: we setup some buckets, groups and privileges, and setup our instances

In this final part we put our backup jobs in place and wrap-up the series. We have covered a lot of CloudFormation syntax and have realy exercised the tool, after this last part we’ll have everything you need to support your next project. If you want to browse through the complete template, it is available on GitHub.

Setting Up Lambda for Snapshotting (and Retention)

What, when and how to backup your instances and data is a complicated subject and covers a lot of ground. For this project we’re only going to deal with the bare minimum: performing nightly snapshots of your images and retaining those snapshots according to a schedule. This will get a reliable image of your server’s storage at backup time but it may not guarantee that your database files are in a consistent state. We don’t walk through the process here, but my advice is to setup a job on your database server that writes a backup of the database to disk before the snapshot job runs. If you ever restore a snapshot you will have a consistent backup of the database right there along with it.

We will use the Amazon Lambda service to perform the snapshot and cull old snapshot images. Since the Lambda script run independent of our instances we can be sure it will always be run and we can monitor the success or failure of the backup job on it’s own: we don’t need to provide administrative access to our instances in order to perform or monitor the backup process. It’s also really nice that we can get the backup job writtent and start it running all from within our CloudFormation script.

Daily Snapshot Job

The first thing we need to do is to setup a new role for our backup job, this role will have the ability to create snapshots, list the existing snapshots, tag and otherwise modify snapshots and write logs to CloudFormation (so we can monitor those backup jobs).

  TutorialLambdaBackupRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: TutorialLambdaBackupRolePolicy
          PolicyDocument:
           Statement:
             - Effect: Allow
               Action:
                 - ec2:CreateTags
                 - ec2:CreateSnapshot
                 - ec2:DeleteSnapshot
                 - ec2:Describe*
                 - ec2:ModifySnapshotAttribute
                 - ec2:ResetSnapshotAttribute
                 - xray:PutTraceSegments
                 - xray:PutTelemetryRecords
                 - xray:GetSamplingRules
                 - xray:GetSamplingTargets
                 - xray:GetSamplingStatisticSummaries
                 - logs:*
               Resource: "*"

We’ve seen all of this stuff before: we create a new role and we associate the AWS service (in this case Lambda) and the AssumeRole action, letting whoever we assign the role the ability to “assume” it. We then create a new policy for the role that allows whoever assumes this role access to manage snapshots for our account as well as the ability to write to the CloudFormation logs and the Amazon X-Ray (tracing) service.

Next we create our Lambda process, I’m going to leave out the code for now and we’ll go over that next. There are better ways to get your backup script into your Lambda but they all involve adding a lot of complexity to this article. When you have the time and energy you can look into this on your own (or maybe I’ll write another article). In any case, having the script inline works for now.

  TutorialCreateBackupLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: ebs-snapshots-create
      Description: Create EBS Snapshots
      Handler: index.handler
      Role: !GetAtt TutorialLambdaBackupRole.Arn
      Environment:
        Variables:
          REGIONS: !Ref AWS::Region
      Runtime: python3.6
      Timeout: 90
      TracingConfig:
        Mode: Active
      Code:
        ZipFile: |
        ...
        code ellided for clarity
        ...

We use the AWS::Lambda::Function resource to provision our new Lambda function ad we assign it a name, description and tell it how to invoke itself. We’re going to write the backup job in Python because it keeps the code simple and nearly everyone knows Python. ;- And next, the Python code itself.

import os
from datetime import datetime, timedelta
import boto3

def handler(event, context):

  ec2_client = boto3.client("ec2")
  env_regions = os.getenv("REGIONS", None)
  today = datetime.today()
  total_created = 0

  if env_regions:
    regions = ec2_client.describe_regions(RegionNames = env_regions.split(","))
  else:
    return total_created

  # loop through all of our regions
  for region in regions.get("Regions", []):
    regionName = region["RegionName"]
    print("Checking for volumes in region %s" % (regionName))
    ec2_client = boto3.client("ec2", region_name = region["RegionName"])

    # query for volumes with matching tags that are in use
    result = ec2_client.describe_volumes(Filters = [
      {"Name": "status", "Values": ["in-use"]}
    ])

    if not result:
      print("No matching EBS volumes with tag 'Client' = '%s' and tag 'Project' = '%s' in region %s" % (clientTag, projectTag, regionName))
      return total_created

    for volume in result["Volumes"]:
      volume_name = volume["VolumeId"]
      volume_description = "Created by ebs-snapshots-create"

      # check the volume for matching tags
      if "Tags" in volume:
        for tag in volume["Tags"]:
          if tag["Key"] == "Name":
            volume_name = tag["Value"]
          elif tag["Key"] == "Description":
            volume_description = tag["Value"]

      # daily
      print("Creating snapshot for EBS volume %s" % (volume["VolumeId"]))
      result = ec2_client.create_snapshot(
        VolumeId = volume["VolumeId"],
        Description = volume_description
      )
      ec2_resource = boto3.resource("ec2", region_name = region["RegionName"])
      snapshot_resource = ec2_resource.Snapshot(result["SnapshotId"])
      snapshot_resource.create_tags(Tags = [
        {"Key": "Name", "Value": volume_name},
        {"Key": "Description", "Value": volume_description},
        {"Key": "Interval", "Value": "Daily"}
      ])
      total_created += 1

      # quarterly
      if today.day == 1 and today.month % 4 == 0:
        print("Creating quarterly snapshot for EBS volume %s" % (volume["VolumeId"]))
        result = ec2_client.create_snapshot(
        VolumeId = volume["VolumeId"],
        Description = volume_description
        )
        ec2_resource = boto3.resource("ec2", region_name = region["RegionName"])
        snapshot_resource = ec2_resource.Snapshot(result["SnapshotId"])
        snapshot_resource.create_tags(Tags = [
          {"Key": "Name", "Value": volume_name},
          {"Key": "Description", "Value": volume_description},
          {"Key": "Interval", "Value": "Quarterly"}
        ])
        total_created += 1

      # annual
      if today.day == 31 and today.month == 12:
        print("Creating annual snapshot for EBS volume %s" % (volume["VolumeId"]))
        result = ec2_client.create_snapshot(
        VolumeId = volume["VolumeId"],
        Description = volume_description
        )
        ec2_resource = boto3.resource("ec2", region_name = region["RegionName"])
        snapshot_resource = ec2_resource.Snapshot(result["SnapshotId"])
        snapshot_resource.create_tags(Tags = [
          {"Key": "Name", "Value": volume_name},
          {"Key": "Description", "Value": volume_description},
          {"Key": "Interval", "Value": "Annual"},
        ])
        total_created += 1

  return total_created

You only have so many characters available for an inline Lambda script so I’ve kept the comments to a minimum. The script is straightforward and the code (for the sake of clarity) doesn’t do anything tricky; it should be easy to follow. Basically what it does is check all of the regions we provided in the REGIONS environment variable and then it checks that region for instances. When it finds one it gets a handle on all of it’s volumes and then snapshots each one.

First the daily snapshot is created, the script always takes this snapshot
If this is the first day of the first month of the quarter then a quarterly snapshot is taken as well
If it’s the last day of the year then we take the annual snapshot

For each snapshot created our script sets the snapshot name to match the volume name and the description of “Created by ebs-snapshot-create” (the name of our Lambda function). It also sets the “Client” and “Project” tags on each snapshot to match the environment variable values.

The script keeps track of how many snapshot it’s created and returns that number when the function ends. Along the way we also print data to standard out: the standard out and the return value will all be gathered and logged when the script runs.

Daily Snapshot Retention Job

We don’t want to accrue snapshots indefininitely, don’t forget, we get charged for their storage. ;-) This next job will enforce our retention policy.

  TutorialDeleteBackupLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: ebs-snapshots-delete
      Description: Delete Old EBS Snapshots
      Handler: index.handler
      Role: !GetAtt TutorialLambdaBackupRole.Arn
      Environment:
        Variables:
          REGIONS: !Ref AWS::Region
      Runtime: python3.6
      Timeout: 90
      TracingConfig:
        Mode: Active
      Code:
        ZipFile: |
        ...
        code elided for clarity
        ...

It’s almost exactly like the last job, aside from it’s name and description. The real difference is in the script for the Lambda function. The script calculates the cutoff date for our daily snapshots: seven days old. As the script checks through the snapshots for the volumes it will check the tags that we placed on the snapshots in the backup job, specificaly the “Interval” tag. Daily snapshots older than our cutoff date will be removed leaving only the seven momst recent daily snapshots.

We do the same for the quartlerly snapshots; we calculate the cutoff data and then check through all of the snapshots tagged with an “Interval” value of “Quarterly”. All of the snapshots past the cutoff are deleted leaving us with the four most recent quartlery snapshots.

The annual snapshots we leave as is, we will accrue those at the rate of once a year forever. You never know when someone will want to dig up old, historic data.

import os
from datetime import datetime, timedelta
from dateutil import relativedelta
import boto3

def handler(event, context):

  ec2_client = boto3.client("ec2")
  date_format = "%Y-%m-%dT%H:%M:%S.%fZ"
  daily_cutoff = (datetime.utcnow() - timedelta(days = 7)).strftime(date_format)
  env_regions = os.getenv("REGIONS", None)
  today = datetime.today()
  total_deleted = 0

  if env_regions:
    regions = ec2_client.describe_regions(RegionNames = env_regions.split(","))
  else:
    return total_created

  for region in regions.get("Regions", []):
    regionName = region["RegionName"]
    print("Checking for snapshots in region %s" % (regionName))

    # query for Daily snapshots with matching tags
    result = ec2_client.describe_snapshots(Filters = [
      {"Name": "tag:Interval", "Values": ["Daily"]}
    ])

    if not result:
      print("No matching Daily snapshots with tag 'Client' = '%s' and tag 'Project' = '%s' in region %s" % (clientTag, projectTag, regionName))
      return total_deleted

    for snapshot in result["Snapshots"]:
      snapshot_time = snapshot["StartTime"].strftime(date_format)
      snapshot_id = snapshot["SnapshotId"]

      if daily_cutoff > snapshot_time:
        snapshot_name = ""
        if "Tags" in snapshot:
          for tag in snapshot["Tags"]:
              if tag["Key"] == "Name":
                  snapshot_name = tag["Value"]

        print("Deleting Daily EBS snapshot ID = %s, Name = %s" % (snapshot_id, snapshot_name))
        ec2_client.delete_snapshot(SnapshotId = snapshot_id)
        total_deleted += 1

    # query for quarterly snapshots with matching tags
    if today.day == 1 and today.month % 4 == 1:
      quarterly_cutoff_raw = datetime.utcnow() - relativedelta.relativedelta(months = 15) # keep five quarters
      quarterly_cutoff = quarterly_cutoff_raw.strftime(date_format)
      result = ec2_client.describe_snapshots(Filters = [
        {"Name": "tag:Interval", "Values": ["Quarterly"]}
      ])

      if not result:
        print("No matching Quarterly snapshots with tag 'Client' = '%s' and tag 'Project' = '%s' in region %s" % (clientTag, projectTag, regionName))
        return total_deleted

      for snapshot in result["Snapshots"]:
        snapshot_time = snapshot["StartTime"].strftime(date_format)
        snapshot_id = snapshot["SnapshotId"]

        if quarterly_cutoff > snapshot_time:
          snapshot_name = ""
          if "Tags" in snapshot:
            for tag in snapshot["Tags"]:
                if tag["Key"] == "Name":
                    snapshot_name = tag["Value"]

          print("Deleting Quarterly EBS snapshot ID = %s, Name = %s" % (snapshot_id, snapshot_name))
          ec2_client.delete_snapshot(SnapshotId = snapshot_id)
          total_deleted += 1

  return total_deleted

With these two functions added to your CloudFormation template you are nearly ready to go. All we need to do is call these jobs every night.

Rules to Invoke our Jobs

We want to run our jobs every night, first the backup job and then the retention job that culls out old snapshots. Lucky for us CloudWatch will let you create a “rule” that is triggered on a schedule, we can create our own rule that will start our backup jobs. The AWS::Events::Rule can be used to do exactly this.

  TutorialPerformBackupRule:
    Type: AWS::Events::Rule
    Properties:
      Description: Triggers our backup lambda script
      ScheduleExpression: cron(30 4 * * ? *)
      State: ENABLED
      Targets:
        - Arn: !GetAtt TutorialCreateBackupLambda.Arn
          Id: TutorialBackupRule

  TutorialDeleteBackupRule:
    Type: AWS::Events::Rule
    Properties:
      Description: Triggers our backup delete lambda script
      ScheduleExpression: cron(45 4 * * ? *)
      State: ENABLED
      Targets:
        - Arn: !GetAtt TutorialDeleteBackupLambda.Arn
          Id: TutorialBackupRule

These two jobs use the familiar cron style syntax, we backup at 4:00AM and then delete old snapshots at 4:45AM every day. Each rule has a pointer to the ARN of the Lambda function it will invokce.

With our jobs all setup, we need to provide our rules with permissions to invoke our Lambda functions with the AWS::Lambda::Permission resource.

  TutorialPermissionCreate:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt TutorialCreateBackupLambda.Arn
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt TutorialPerformBackupRule.Arn

  TutorialPermissionDelete:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt TutorialDeleteBackupLambda.Arn
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt TutorialDeleteBackupRule.Arn

We add the InvokeFunction permission to our two CloudFront triggers so that they are allowed to invoke our Lambda functions. With that our backups are all taken care of.

Template Wrap-Up: Output

The last bit of any CloudFront template is provided output data, this data will be provided as a JSON object. You could provide thiss data to another process that double-checks that the resources have been created or performs additional tasks. Here’s our output section…

Outputs:
  WebServer:
    Description: A reference to the web server
    Value: !Ref TutorialWebServer

  DatabaseServer:
    Description: A reference to the database server
    Value: !Ref TutorialDatabaseServer

  BackupS3Bucket:
    Description: S3 Bucket used to backup data
    Value: !Sub |
      https://${TutorialBackupS3Bucket.DomainName}\

Here we return references to our two server instances and our S3 bucket. There are certainly more items we could list in this section, perhaps our reference to our VPC. For now we’ll leave it short and to the point.

Whew! That is the last piece in the series! You can model on the template we’ve put together here and start provisioning resources on CloudFormation with a slightly more rapidity and confidence. ;-D

Provision Pragmatically and Programmatically with CloudFormation, 2

2019-02-05T00:00:00-05:00

This is the second part in a three part series where we build up a CloudFormation template for what I think of as a pretty typical environment: one virtual private cloud broken into two subnets and including two instances. If you haven’t read the first part in the series, I encourage you to check it out now!

Setup an “Admininstrators” Group

For this project we’re creating a general “administrators” group that has access to the backup bucket with our database backups. I generally don’t allow anyone direct access to the EC2 console unless they really need it, so I don’t address that in this template (in addition, not every EC2 web console action can be managed this way). My thinking here is that we typically give “administrators” or “developers” accounts on the individual instances, they don’t really need the EC2 web console in order to get their work done.

  TutorialAdminGroup:
    Type: AWS::IAM::Group
    Properties:
      Policies:
        - PolicyName: TutorialAdminPolicy
          PolicyDocument:
            Statement:
            - Effect: Allow
              Action:
                - s3:ListBucket
                - s3:ListBucketByTags
                - s3:ListBucketMultipartUploads
                - s3:ListBucketVersions
              Resource: !Sub ${TutorialBackupS3Bucket.Arn}
            - Effect: Allow
              Action:
                - s3:PutObject
                - s3:DeleteObject
                - s3:GetObject
              Resource: !Sub ${TutorialBackupS3Bucket.Arn}/*

We use the IAM Group resource to create our new group. The only property that we provide is the Policies property that contains a list of PolicyName and PolicyDocument pairs, in this case we define only one that we have named “TutorialAdminPolicy”.

A PolicyDocumentcontains a Statement that holds a list of Effect instances; each of those in turn contains an Action property with a list of permissions. We use the Resource property to tie in a reference to our bucket, in this case the backup bucket’s ARN. If we take a look at the first Effect, you’ll see that we’ve assigned four “ListBucket…” permissions for our backup bucket. The second effect applied three more actions to every file in the backup bucket, that’s indicated by the /* at the end of the bucket’s ARN.

Any account that we add to this group will be able to download or delete any of the files in the bucket (as well as upload). In this case when we say “files” we really mean database dumps for this project. They will also be able to pull up the S3 console for the bucket but they will need a link that is directly to the bucket. They will not be able to log into S3 and browse the list of all buckets.

One last note: since we are creating an IAM role with our template, we need to let CloudFormation know that this is okay. From here on out we need to add the --capabilities flag to our aws commands.

aws cloudformation create-stack \
  --stack-name tutorial \
  --template-body file://template.yaml \
  --parameters ParameterKey=KeyPairName,ParameterValue=cloudfront-tutorial \
  --tags Key=Project,Value=cf-tutorial \
  --capabilities CAPABILITY_IAM

If you don’t include the tag, CloudFormation will exit with an error. The same flag can be used with “update” commands.

Setup a Role for Our Instances

Next we need to setup the role that our instances can assume to get access to resources (for now, just the backup bucket).

  TutorialInstanceRole:
    Type: AWS::IAM::Role
    Properties:
      ManagedPolicyArns:
        - arn:aws-us-gov:iam::aws:policy/CloudWatchAgentServerPolicy
        - arn:aws-us-gov:iam::aws:policy/service-role/AmazonEC2RoleforSSM
        - arn:aws-us-gov:iam::aws:policy/AmazonSSMReadOnlyAccess
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole

We use the Role resource to create our new instance role and we assign three canned policies…

the first lets the instance run the CloudWatch agent and submit events to the CloudWatch service
the next lets the instance interact with Systems Manager, letting us manage the instances as a group
The last provides read-only access to the Systems Manager parameter store.

Amazon’s CloudWatch service will aggregate data submitted by your instances and let you setup reports, dashboards and alerts based on this data (i.e. when CPU usage gets too high or available disk space is too low). Systems Manager provides some tools for managing your instances, like installing patches or a software package. It lets you perform these actions on several instances as a group which can be handy. I’m not going to go over these services in-depth here but I encourage you to spend some time checking them out if you haven’t done so already.

  TutorialInstanceRolePolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: TutorialInstanceRolePolicy
      Roles:
        - Ref: TutorialInstanceRole
      PolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - s3:ListBucket
              - s3:ListBucketByTags
              - s3:ListBucketMultipartUploads
              - s3:ListBucketVersions
            Resource: !Sub ${TutorialBackupS3Bucket.Arn}
          - Effect: Allow
            Action:
              - s3:PutObject
              - s3:DeleteObject
              - s3:GetObject
            Resource: !Sub ${TutorialBackupS3Bucket.Arn}/*

We are creating a new Policy resource that we will assign to our instances. We link the role we just created and then add a new policy that provides access to the backup bucket. As you can see, it is exactly the same as the role that we created for our administrators group.

The last piece of this puzzle is an “instance profile” that we can assign to our instances.

  TutorialInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Roles:
        - !Ref TutorialInstanceRole

With all of the other work done, there’s not much to see here. We create a new InstanceProfile resource and link it to our role. When we provision instances we can assign them this profile, they will then have the ability to write to our backup bucket, submit performance and event data to CloudWatch and accept remote commands from System Manager.

Provision CloudWatch Log Groups

The very last thing we need before provisioning are a couple CloudWatch log groups. We can instruct our instances to ship some of their log files out to these groups and then view the logs through the CloudWatch console.

  TutorialBootLog:
    Type: AWS::Logs::LogGroup

  TutorialKernelLog:
    Type: AWS::Logs::LogGroup

We use the LogGroup resource to create two new log groups. These groups will be names after the name we provide in the template with some random characters tacked on at the end. We’re creating the groups now because we’re going to reference them directly when we provision the instances.

Provision the Database Server

I know what you are thinking: all that work and we’re only now provisioning our instances! It’s true, there was a lot of preparation and things to think about but we are well on our way to having a template that will provision our servers in a repeatable and reasonably safe manner. Just think of all the templates you will be writing!

We’re going to provision the database server first. This one is a little bit involved so I’m going to break it up into pieces. First we provision a new instance with the Instance resource…

  TutorialDatabaseServer:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        config:
          files:
            /etc/profile.d/cloudformation-init.sh:
              content: !Sub |
                export BACKUP_S3_BUCKET="${TutorialBackupS3Bucket}"

Note that in the code stanza above I stop before the properties, we’ll go over the properties next.

The first thing we do is setup the “metadata” for the CloudFormation initialization script (we will call that at the very end) with the CouldFormation Init type. This is were we tell the initialization script what we would like it to do when it runs, in this case we add a new environment variable to a new file in /etc/profile.d called cloudformation-init.sh. Whenever someone logs into the machine this script (along with everything else in the directory) will be evaluated, the end result will be that the name of our backup bucket will be available through the BACKUP_S3_BUCKET environment variable. Keep in mind that this metadata doesn’t do anything on it’s own: the initialization script will use it when it runs and we’ll do that at the end of our instance definition.

We also need to provide the configuration file for the CloudWatch agent. This agent will collect performance data and log files and ship them back to the CloudWatch service.

            /etc/amazon/amazon-cloudwatch-agent.json:
              content: !Sub |
                {
                  "metrics": {
                    "append_dimensions": {
                      "AutoScalingGroupName": "${!aws:AutoScalingGroupName}",
                      "ImageId": "${!aws:ImageId}",
                      "InstanceId": "${!aws:InstanceId}",
                      "InstanceType": "${!aws:InstanceType}"
                    },
                    "metrics_collected": {
                      "cpu": {
                        "measurement": [
                          "cpu_usage_idle",
                          "cpu_usage_iowait",
                          "cpu_usage_user",
                          "cpu_usage_system"
                        ],
                        "metrics_collection_interval": 30,
                        "totalcpu": false
                      },
                      "disk": {
                        "measurement": [
                          "used_percent",
                          "inodes_free"
                        ],
                        "metrics_collection_interval": 30,
                        "resources": [
                          "*"
                        ]
                      },
                      "diskio": {
                        "measurement": [
                          "io_time"
                        ],
                        "metrics_collection_interval": 30,
                        "resources": [
                          "*"
                        ]
                      },
                      "mem": {
                        "measurement": [
                          "mem_used_percent"
                        ],
                        "metrics_collection_interval": 30
                      },
                      "statsd": {
                        "metrics_aggregation_interval": 30,
                        "metrics_collection_interval": 10,
                        "service_address": ":8125"
                      },
                      "swap": {
                        "measurement": [
                          "swap_used_percent"
                        ],
                        "metrics_collection_interval": 30
                      }
                    }
                  },
                  "logs": {
                    "logs_collected": {
                      "files": {
                        "collect_list": [
                          {
                            "log_group_name": "${TutorialBootLog}",
                            "file_path": "/var/log/boot.log"
                          },
                          {
                            "log_group_name": "${TutorialKernelLog}",
                            "file_path": "/var/log/messages"
                          }
                        ]
                      }
                    }
                  }
                }

This configuration file tells the CloudWatch agent to ship back data on CPU, disk, memory and swap file usage back to CloudWatch as well as two log files: the boot and kernel logs.

Next we’ll set the properties for our instance…

    Properties:
      ImageId: !Ref ImageId
      InstanceType: !Ref InstanceType
      KeyName: !Ref KeyPairName
      NetworkInterfaces:
        - SubnetId: !Ref TutorialPrivateSubnet
          DeviceIndex: 0
          GroupSet:
            - !Ref TutorialPrivateSecurityGroup
      BlockDeviceMappings:
        - DeviceName: !Sub "/dev/${RootDevice}"
          Ebs:
            VolumeSize: !Ref VolumeSize
            VolumeType: gp2
      IamInstanceProfile: !Ref TutorialInstanceProfile
      Tags:
        - Key: "Name"
          Value: "Tutorial Database Server"

Way back in part 1 we create these parameters and provided default values for them, now they are finally coming into play. If you look back at the top of the template, you’ll see that we set parameters for the image, instance type, key pair name and root volume. While our template requires that a key pair name be provided the other values are populated with default values. My expectation is that the default values will normally be used, most of the reason they are at the top in the parameter section is to make them easy to get at.

The first thing we do is choose the image we’d like for this instance, in the code above we have a reference to the parameter, the default value is for the Amazon Linux 2 image. There are other Linux images out there, I picked this one mostly because it already has the AWS tools and CloudFormation stuff installed, making things that much easier. Also kind of interesting, if you develop locally you can work with an Amazon Linux 2 Docker Container on your workstation.

Next we use our instance type parameter, the default values is for t2.micro because this is a tutorial and I don’t want to cost you money; this type is eligible for use under Amazon’s free tier. If you haven’t used up your 750 hours of free tier usage for the month you shouldn’t be charged for provisioning these instances. Setting cost aside, a micro instance is likely enough to host a low traffic website, like your personal blog.

We set the name of the key pair we want to use to provision our instance, note that the value we reference in the KeyName property is the one parameter we set for this script back in part 1. You will need to have this key pair handy in order to SSH to the instance.

We’d like the database server to be on our private subnet and we take care of that when we specify the NetworkInterfaces property. Here we pass in a reference to our private subnet, we then set the security group to our “private” security group with the GroupSet property of the network interface.

Every instance needs disk space and we can customize the Elastic Block Store (EBS) volumes for our instance with the BlockDeviceMappings property. We map in one volume of 250GB (referencing our parameter) to the root device by using the substitution function with our parameter to set the value to /dev/xvda, we chose the “general purpose” (gp2) volume type.

The default volume type is “gp2” and it’s a reasonable choice, more information about the various types may be found in the EBS volume type documentation. Also note that the root device that the instance boots from may vary from one Linux distribution to the other. For instance, under Ubuntu the root volume needs to be mapped to /dev/sda1; if the instance can’t find the volume you will see it start up in the EC2 console but it will stop in just a couple of minutes.

Instead of managing credentials for our EC2 instances we thoughtfully created a profile! We set the profile for our instance with the IamInstanceProfile property and pass in a reference to that profile.

Lastly we set tags on our instance, in this case we set the “Name” tag.

With that out of the way the only thing left to do is to invoke the CloudFormation initialization script when the instance starts up. To get this done we write a little script that calls cfn-init at start time.

      UserData: 
        Fn::Base64: 
          !Sub |
            #!/bin/bash -xe

            # cloudformation initialize
            /opt/aws/bin/cfn-init -v -s ${AWS::StackName} --region ${AWS::Region} -r TutorialDatabaseServer

            # download and install cloudwatch agent
            wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
            yum -y install amazon-cloudwatch-agent.rpm
            mv /etc/amazon/amazon-cloudwatch-agent.json /etc/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.json
            sudo systemctl enable amazon-cloudwatch-agent
            sudo systemctl start amazon-cloudwatch-agent

Documentation for cfn-init is available on the CloudFormation site. You can see in the example above that we call the script with the name of our stack, the stack’s deployment region and the name of our server. When the script runs it will inspect the Metadata property that we set at the the beginning of our instance stanza and will carry out those tasks. For this instance, the initialization script will add that new file to /etc/profile.d with our custom environment variables.

The cfn-init script handles getting our configuration files written to disk, the other thing we need to do is get the CloudWatch agent installed and running. The next bit of script downloads the current version, installs it, moves our configuration file into place and then enables and starts the service so that the agent is started at boot time.

And that’s it… For our database server. We need to do pretty much the same thing for our web server.

  TutorialWebServer:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        config:
          files:
            /etc/profile.d/cloudformation-init.sh:
              content: !Sub |
                export BACKUP_S3_BUCKET="${TutorialBackupS3Bucket}"
                export DATABASE_SERVER="${TutorialDatabaseServer.PrivateIp}"

Mostly everything is exactly the same but there are a couple small differences. When we setup the Metadata for the initialization script we added another environment variable that contains the private IP address of the database server, in this way we can deploy this template more than once and the web server in each stack will know where to find it’s matching database server.

I don’t include it in this article, but we provide the exact same configuration file for the CloudWatch agent that we used for the web server. If you are writing the template along with this article, take a minute to copy and paste that stanza into the file now.

    Properties:
      ImageId: !Ref ImageId
      InstanceType: !Ref InstanceType
      KeyName: !Ref KeyPairName
      NetworkInterfaces:
        - SubnetId: !Ref TutorialPublicSubnet
          DeviceIndex: 0
          GroupSet:
            - !Ref TutorialPublicSecurityGroup
      BlockDeviceMappings:
        - DeviceName: !Sub "/dev/${RootDevice}"
          Ebs:
            VolumeSize: !Ref VolumeSize
            VolumeType: gp2
      IamInstanceProfile: !Ref TutorialInstanceProfile
      Tags:
        - Key: "Name"
          Value: "Tutorial Web Server"
      UserData: 
        Fn::Base64: 
          !Sub |
            #!/bin/bash -xe

            # cloudformation initialize
            /opt/aws/bin/cfn-init -v -s ${AWS::StackName} --region ${AWS::Region} -r TutorialDatabaseServer

            # download and install cloudwatch agent
            wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
            yum -y install amazon-cloudwatch-agent.rpm
            mv /etc/amazon/amazon-cloudwatch-agent.json /etc/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.json
            sudo systemctl enable amazon-cloudwatch-agent
            sudo systemctl start amazon-cloudwatch-agent

In terms of the properties, the only difference is that we have placed our web server in the public subnet so that it can communicate with the public internet (as web servers so often need to do).

At this point we have covered the first four goals that we laid out for ourselves in part 1. With the template as it stands right now, we can…

Provision resources for a project
Retire resources for a project
Isolate resources for a project
Report on resources by project

I think this is a pretty significant milestone! From here we can tweak the template to match a particular project and with just one command setup the environment. If we have a client that wants a “test” and a “production” environment, all we need to do is provision the template twice.

Reporting on Resources

This is a little outside the realm of CloudFormation, but I thought we’d take a short break from the template and take a look at some ways we can use the tags we have so diligently been placing on our resources. Take a look at the billing console for your account. You will be dropped at the dashboard and some summary data for your account will be visible.

If you haven’t done so already, click on the “Cost Explorer” link from the left-hand navigation bar and enable it for your account. It’s a handy tool that lets you do some simple querying and reporting on your resources and their costs.

Next click on the “Cost Allocation Tags” link, also on the left-hand navigation bar. At the top you can see that there are some cost allocation tags that AWS can generate on it’s own, go ahead and click the “Activate” button to turn those on. Lower down on the page will be a list of all of the tags that we have defined (as well as any tags you had already setup). You can choose which tags you want to make available in the Cost Explorer but my advice is just to click the top-most checkbox and make them all active. You never know when a tag will come in handy! Click the “Activate” button and confirm that, yes, you want the tags to be “active”.

Now click back to the “Cost Explorer” link and click on “Launch Cost Explorer”. If you just activated the cost explorer now, you will probably have to wait until tomorrow; keep it in mind and try to come back to this section.

If the Cost Explorer has data on hand then you will be presented with another dashboard attempting to summarize your costs on amazon. Click on the magnifying glass icon on the left-hand navigation bar and choose “Cost and Usage”, the report builder will appear with your last six months of spending. Click on the “Last 6 Months” pull-down and choose “1M” from the “Historical” section at the bottom, this will show your last month of spending.

On the right-hand side is a list of filters and this is where the tags will come in handy. Click on the “Tag” link in the filter list, a list of your tags will appear; click on “Project” and a list of all your values for the “Project” tag will be listed. In this tutorial we’ve been putting “cf-tutorial” in the “Project” tags, check the box for “cf-tutorial” and then press the “Apply filters” button to update the report.

What you are looking at now is likely a very dull report, because we’ve been using low-cost free-tier instances. But, still, what we have is a report on just the resources that belong to this project. If you were to place a “Client” tag in your templates you could report on the entire cost of a client (maybe you could use that to figure out how to bill) and you can break down a client’s costs by particular projects. It is a valuable tool and definitely worth putting some time in exploring your options.

Set Alarms on Instance Usage

I think of this as related to our reporting goals: we’d like to set some alarms that monitor the data we’re sending to CloudWatch to let us know if our servers start to go off the rails. We’ll set alarms for when CPU usage gets high or if we start to run out of disk space but I bet you can think of lots of other things you’d like to monitor.

We’re going to use the Simple Notification Service to alert us when an alarm is hit. The way this will work is that when CloudWatch sees an alarm trigger, it will send a message to the SNS “topic”. It will be our responsibility to log into the SNS console and add ourselves to the notification list (by email or text message), we aren’t going to populate the list of subscribers.

  TutorialAlarmTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: TutorialAlarms

We use the Topic resource to create our new alarm target topic. With the SNS topic created we can now create our first alarm.

  TutorialDatabaseCPUAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: Database Server CPU Usage High
      AlarmActions:
        - !Ref TutorialAlarmTopic
      MetricName: CPUUtilization
      Namespace: AWS/EC2
      Statistic: Average
      Period: 60
      EvaluationPeriods: 3
      Threshold: 89
      ComparisonOperator: GreaterThanThreshold
      Dimensions:
        - Name: InstanceId
          Value: !Ref TutorialDatabaseServer

With the Alarm resource we create an alarm for CPU usage on the database server, if the server is using more than 89% of the CPU for more than 3 minutes then the alarm will trigger and send a message to our SNS topic.

  TutorialDatabaseDiskAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: Database Server Disk Usage High
      AlarmActions:
        - !Ref TutorialAlarmTopic
      MetricName: disk_used_percent
      Namespace: CWAgent
      Statistic: Average
      Period: 60
      EvaluationPeriods: 3
      Threshold: 89
      ComparisonOperator: GreaterThanThreshold
      Dimensions:
        - Name: InstanceId
          Value: !Ref TutorialDatabaseServer
        - Name: ImageId
          Value: !Ref ImageId
        - Name: InstanceType
          Value: !Ref InstanceType
        - Name: path
          Value: /
        - Name: device
          Value: !Sub ${RootDevice}1
        - Name: fstype
          Value: !Ref RootFsType

This next alarm is for the amount of disk used by the database server, here we set an alarm to trigger if more than 89% of the disk is in use. This one will also wait for the disk to be in this state for 3 minutes before triggering the alarm.

Provisioning this alarm is a bit more work, you can see that we had to add several Dimensions to get the alarm setup. This is because when the CloudWatch agent reports this data it references all of these dimensions and we need to match them all in order to have a working alarm. If we had left one of these out then the alarm wouldn’t match any data and would never trigger.

The only bit left is the backup jobs but we’ll leave that for part 3. ;-) Congratulations on working your way through all of this material! While it can be pretty dry reading you can really get a lot of mileage building templates for projects that you know you will be deploying over and over.

Provision Pragmatically and Programmatically with CloudFormation

2019-01-25T00:00:00-05:00

How many times have you gazed upon the list of services that Amazon is providing through their Amazon Web Services (AWS) family of products and felt an unpleasant mixture of anxiety and confusion? If you’re anything like me, it would be literally every time you visited their website. While I was one of the first in my circle of software developer friends to get on board the Elastic Compute Cloud (EC2) train, I haven’t been great at keeping up with new developments. This year, I decided, would be the year that I get back on track and sort out this alphabet soup of cloud virtualization technology.

After clicking around at random and reading an unwholesome amount of bland introductory pages, I finally started to get an idea of how I should try to put all of these pieces together. The first thing I figured out:

It is way, way too hard to get everything up and running by clicking a bunch of buttons.

What are your goals? For me, it’s to be able to deploy a web-based application for one of my clients. As a corollary, I want to be able to entirely scrap a web-based application for one of my clients (maybe it’s been replaced, maybe they’ve hired another consultant). In effect, that’s two goals. Plus I would like to keep all of my clients separate, I don’t want changes to a firewall rule to effect all of my clients, just the one client who’s willing to take the risk. If you think about it, keeping things isolated will also make it easier to figure out who we can charge for what services at the end of the month; we should also tag these resources in a way that makes sense. That’s four goals and that’s only what we’ve thought of so far. And we should snapshot on a daily basis and maybe not keep all of the snapshots. Oh, backing up the database is important, too!

Provision resources for a project
Retire resources for a project
Isolate resources for a project
Report on resources by project
Snapshot instance volumes and delete old backups on a schedule
Backup the databases on a schedule and delete old backups on a schedule

Looking at these goals, the number of services, the way many of them build on each other… It all becomes clear that the answer is automation. Of course, Amazon provides an automation tool: CloudFormation.

This is a first in a series of articles that will cover all of these goals. We’ll start out with an empty file and we’ll gradually build that file up, adding more and more resources to that file as we go along. I am planning three articles all together, this on covers a brief introduction to the tools and gets as far as setting up a virtual private cloud for the project, splitting it into two networks and putting security groups in place. So exciting!

I encourage you to follow along as we build up the template but you can always checkout the finished product over at GitHub.

Install the AWS Command Line Tools

Amazon provides a goofy drag-and-drop interface for CloudFormation but that runs kind of counter to the whole point of this article. You can also write your CloudFormation template and then upload it via their web interface, I think that’s also a little clunky. My recommendation is that you install the AWS Command Line Tools and use those tools to deploy. Amazon also provides packages for Windows, but if you’re working from within the Windows environment you might be more interested in the AWS Tools for PowerShell which do much the same thing but, you know, with PowerShell.

This is kind of a power tip, but if you’re managing more than one Amazon account or dealing with more than one region, you can configure profiles to make that process easier.

With one of these tools installed and properly configured, you can invoke CloudFormation directly from your console window! :-)

Start Our New CloudFormation Template

In the world of CloudFormation, the document you are writing is called a “template”. I believe it’s referred to as a template because one template can be used to create many different sets of resources. For instance in this tutorial we’re going to be writing a template that will provision a web server and a database server; we can use the template to provision as many of these web-and-database-server pairs as we might like, all independent of one another.

You can write your template in YAML or JSON, if you’re writing it by hand (as we are right now) then I strongly recommend that you write it in YAML. I know, I know, YAML isn’t everyone’s favorite but getting all of the ending braces and brackets lined up in a JSON document is it’s own kind of mental torture. You deserve better, trust me: don’t do that to yourself.

Anyway, we’re going to start off with a description and a parameter. A parameter, as you may have suspected, is a value that we can supply when we are actually using our template to provision something. Open up a new file in your favorite editor, type in the text below and save it as template.yaml.

Description: Provisions a simple cluster with a web and a database server

Parameters:
  KeyPairName:
    Type: String
    Description: Name of keypair used when provisioning instances

Throughout this article we’ll continue to build up this template but if you need to jump ahead or need to refer to the entire thing, you can browse through it in the GitHub project.

Some decisions are best postponed until the absolute last minute and parameters are here when you have decisions to postpone. When we decide to provision our resources, we can provide the name of the EC2 key pair to use at that time.

I’m not going to go into a big discussion about how to manage your EC2 key pairs. Maybe you have one per person who has access to provision instances. Maybe you have one per client. However you manage it, you can pass it in when you provision resources.

Next we’re going to add some more parameters but we’ll be providing default values for all of them. The thinking here is two-fold: first, we want the flexibility to be able to change some of the features at provisioning time. Second, we can reference these parameters throughout the template and this keeps us from sprinkling these identical values throughout the template.

  InstanceType:
    Type: String
    Description: Instance type to provision
    Default: t2.micro
  ImageId:
    Type: String
    Description: Image to use when provisioning instances
    Default: ami-035be7bafff33b6b6  # Amazon Linux 2
  RootDevice:
    Type: String
    Description: Name of the root device on provisioned instances
    Default: xvda  # varies based on Linux type (Ubunut /dev/sda1)
  RootFsType:
    Type: String
    Description: File system type for root device
    Default: xfs

You can see that these all relate to the instances that we will be provisioning. We call out the instance type, the image we’ll use to create the instances and the name of the root volume and it’s file type. Note that the root volume and file type aren’t something we can really choose, they are dictated by the image (in this case, Amazon Linux 2).

Provision a New Virtual Private Cloud

We need at least one resource in order to provision our template. For this project we’re going to create our own Virtual Private Cloud (VPC) to hold our resources. This will keep things isolated from everything else you might have deployed to AWS.

Resources:

  TutorialVPC:                   # name of the resource
    Type: AWS::EC2::VPC          # type of resource
    Properties:                  # properties of the resource
      CidrBlock: "10.6.0.0/24"
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: "Name"
          Value: "Tutorial VPC"

The Resources section of the template will contain a stanza for each of the resources we provision, this will be the bulk of the template. Each stanza in this section will start with the name of the resource (TutorialVPC), we can use this name when we need to reference the resource while creating other resources (for instance, when we add subnets to this VPC). After the name we need to declare the Type of resource: every CloudFormation resource has a type, they are all clearly documented in the CloudFormation API Documentation. In the example above we are creating a new VPC instance. Next we set the Properties for our resource, the values provided here will be particular to each resource. Most resources accept a Tags property but not all, if you set Tags on a resource that doesn’t support that property CloudFormation will throw an error.

Every VPC needs a block of IP addresses, we set this with the CidrBlock property and provide a non-routable network with 254 available addresses. I chose a network that won’t conflict with the address range I use at home or at work, this is important if you ever want to use a Virtual Private Network (VPN) to connect your Amazon VPC to your own network, perhaps to make access to these resources easier.

We’d like Amazon to continue to assign names to our instances, so we set the EnableDnsSupport and EnableDnsHostnames properties to true. Lastly we set some tags on our resources to remind ourselves why we provisioned them in the first place.

Provision Resources With Our Template

With our template set, we’re ready to use it to provision some resources! When we provide the template to CloudFormation, we also have to provide our parameter values and we have the opportunity to tag all of the provisioned resources (maybe by client or project or both).

Go ahead and run the command below to begin provisioning. When CloudFormation sets up all of the resources it links them together into a “stack”. Once complete, you can deal with the stack as one unit. The stack name provides a human-friendly handle for all of the related resources.

aws cloudformation create-stack \
  --stack-name tutorial \
  --template-body file://template.yaml \
  --parameters ParameterKey=KeyPairName,ParameterValue=cloudfront-tutorial \
  --tags Key=Project,Value=cf-tutorial

Remember to provide your own key pair name where we have used cloudfront-tutorial.

As soon as CloudFormation receives the template it will return a new StackId with the Amazon Resource Name (ARN) for the stack, you will have to wait a bit for the stack to finish being provisioned. You can check on your stack through the CloudFormation web console and see how things are going. Some resources take longer than others, CloudFormation will ensure they are created in the right order and work out the dependencies. This template should move along pretty quickly as we aren’t provisioning much.

I broke the command over several lines to try and make it easier to follow. In the create-stack command above we…

Ask the AWS tools to call out to CloudFormation with the “create-stack” command
Set the name of the stack we are creating to “tutorial”
Provide our template to CloudFormation with the file:// URL
We set the one template parameter, KeyPairName (note the weird way we have to set parameters)
We provided a one tag key and value that CloudFormation will use to tag all of the provisioned resources

You may separate multiple parameters with a space, multiple tags are separated with a comma. It’s weird, I don’t know why they aren’t uniform.

When provisioning is complete, you can head over to the VPC web console to inspect your new virtual cloud. There’s not a lot to see, but if you choose “Yourq VPCs” from the left-hand navigation bar and then select your “Tutorial VPC” from the list, you can click the “Tags” tab and look at the tags associated with the resource. You’ll see the tag we specified in the template (Name) and the tag we passed into the create-stack command (Project). CloudFormation also set several tags of it’s own, linking the resource to the stack.

As we work through the template we might update the stack or delete it entirely. The command to update the stack is almost exactly the same.

aws cloudformation update-stack \
  --stack-name tutorial \
  --template-body file://template.yaml \
  --parameters ParameterKey=KeyPairName,ParameterValue=cloudfront-tutorial \
  --tags Key=Project,Value=cf-tutorial

If CloudFormation runs into any problems while creating or updating your stack, it will roll back to the last valid state of the stack. If this happens during the creation of the stack you’ll end up with no resources at all, if it happens during an update then it will be as if your update attempt never occurred.

Deleting the stack is quite a bit shorter.

aws cloudformation delete-stack --stack-name tutorial

Go ahead and run that command now to delete the stack. The AWS tool command won’t return any results but if you switch to the CloudFormation console you might catch a glimpse of the stack before it’s entirely deleted. If you switch over to the VPC console you’ll see the new VPC is gone, it was deleted along with the stack.

In my opinion, this is a really nice way to work. You can edit your template and work on getting all of the bits and pieces of your stack created and linked together in a way that makes sense (provisioned in the VPC, in the correct subnet, with the correct security group and policies, etc.) When you feel pretty good about your work you can go ahead and provision. If there’s a problem or something isn’t lining up the way it should you can simply delete the stack and continue to work on your template. For me, this is far more convenient then clicking through the various web consoles and trying to link everything up by filling out fields or applying actions to all of the various pieces.

Retaining Resources through Stack Updates and Even Deletion

As much as CloudFormation can, it will try to preserve resources between updates. On the other hand there are some changes that can’t be made after provisioning: in those cases CloudFormation will delete the resource and create a replacement. For many things this doesn’t really matter, but for things like instances or Elastic IP addresses you may really need to retain those resources. The CloudFormation Documentation is pretty clear on what resource updates can be performed without “service interruption”, keep your eyes open for those warnings.

You can also set a DeletionPolicy for the resources in your template. This lets you signal to CloudFormation that, for certain resources, you would like to keep the resource (even if a replacement is required for provisioning) or (if the resource supports it) perform a “snapshot” backup before deleting the resource (i.e., EC2 volumes or RDS instances). That said, even with the deletion policy set for your critical resources, you should think hard about updating them after provisioning the stack.

My recommendation is that you leave the deletion policies out of your template while you are working on it. Wait until you feel pretty good about the template, maybe until the point where you’re ready to deploy your application or project. When you reach that point, go back and set the deletion policy to Retain on those things that are truly critical (volumes with data on them, instances you’ve customized heavily, etc.) and then update your stack.

Now that we have the basic ideas and the machinery down, we can move a little more quickly.

Provision an Internet Gateway

For this project We’re going to divide our virtual private cloud into two subnets: one will be able to communicate with the public internet and will house our web server. The other subnet will be able to talk to the first but will not face the internet, it will house our database server. In this way we can ensure that machines on the public internet cannot communicate directly with our database server.

In order to get our external subnet facing the internet, we need to provision an Internet Gateway and then setup routes so that instances in our virtual cloud can send and receive traffic over the public internet.

  TutorialInternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: "Name"
          Value: "Internet Gateway"

There isn’t a lot to configure for the InternetGateway resource. We give it a name and add a tag. Next we need to attach it to our virtual cloud.

  TutorialVPCGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref TutorialVPC
      InternetGatewayId: !Ref TutorialInternetGateway

There’s not a lot to configuring the VPCGatewayAttachment either. The interesting bit is where we tell it which VPC to which we’d like it attached. We use the !Ref function to indicate that we are providing a reference to another resource in the template and then we provide the name of that resource.

Setup Our Subnets

With that internet stuff out of the way, we can create our two subnets. First, our public subnet.

  TutorialPublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref TutorialVPC
      CidrBlock: "10.6.0.0/25"
      MapPublicIpOnLaunch: true
      Tags:
        - Key: "Name"
          Value: "Public Subnet"

We use the Subnet resource to split off a chunk of our VPC with half of our address space, we use the VpcId property to link this subnet declaration to our VPC. We set the MapPublicIpOnLaunch property to true so that each instance gets a public IP when launched.

  TutorialPrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref TutorialVPC
      CidrBlock: "10.6.0.128/25"
      MapPublicIpOnLaunch: false
      Tags:
        - Key: "Name"
          Value: "Private Subnet"

In the stanza above, we take the rest of our address space and allocate that to our private subnet, again we use the VpcId property to link this subnet declaration to our VPC. We don’t want instances in this network to have public IP addresses and we indicate that by setting the MapPublicIpOnLaunch property to false.

Setup Our Routing

We always have a default route that lets the addresses in our VPC communicate with each other but we don’t start out with any routes out to our internet gateway and the public internet. In order to setup our routes we first need to setup our routing tables.

  TutorialPublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref TutorialVPC
      Tags:
        - Key: "Name"
          Value: "Public Internet Route Table"

  TutorialPrivateRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref TutorialVPC
      Tags:
        - Key: "Name"
          Value: "Private Route Table"

There’s not much to the RouteTable resource, we indicate that we’re going to be managing routes for our VPC with the VpcId property and set a tag.

Next we’re going setup the actual routes.

  TutorialInternetRoute:
    Type: AWS::EC2::Route
    DependsOn: VPCGatewayAttachment
    Properties:
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref TutorialInternetGateway
      RouteTableId: !Ref TutorialPublicRouteTable

The Route resource adds a new route to a routing table. Here we add a route to our public table to route traffic for addresses on the public internet through our internet gateway. We let CloudFormation know that this resource shouldn’t be created unless the VPCGatewayAttachment is already setup with the DependsOn attribute. Like the Retain attribute, you can place this on any resource.

  TutorialPublicSubnetPublicRoute:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref TutorialPublicRouteTable
      SubnetId: !Ref TutorialPublicSubnet

With our route to the public internet setup, we can use the SubnetRouteTableAssociation resource to link our routing table to our public subnet.

Next we need to do something similar for our private subnet. The big difference here is that our private subnet doesn’t connect directly to the internet. Instead we will use a Network Address Translation (NAT) gateway to let instances on our private subnet connect to the internet without allowing machines on the public internet to initiate connections with instances in our private subnet.

  TutorialNatGatewaySubnetRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref TutorialVPC
      Tags:
        - Key: "Name"
          Value: "NAT Gateway Subnet Route Table"

We create a new routing table and associate it to our VPC.

  TutorialNatGatewayInternetRoute:
    Type: AWS::EC2::Route
    DependsOn: TutorialVPCGatewayAttachment
    Properties:
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref TutorialInternetGateway
      RouteTableId: !Ref TutorialNatGatewaySubnetRouteTable

And then we create a route to the public internet through our internet gateway.

The next step is to create our NAT gateway, but first we need to get a publicly accessible IP address that will be assigned to our NAT gateway. We can use the EIP (Elastic IP) Resource to get a handle on a new Elastic IP address.

  TutorialNatGatewayElasticIP:
    Type: AWS::EC2::EIP
    Properties:
      Domain: vpc

We set the domain value to vpc, indicating that we want to use this address from our VPC. Kind of funny, that’s the only valid value for that property.

Now we can setup our NAT gateway…

  TutorialNatGateway:
    Type: AWS::EC2::NatGateway
    DependsOn: VPCGatewayAttachment
    Properties:
      AllocationId: !Sub ${TutorialNatGatewayElasticIP.AllocationId}
      SubnetId: !Ref TutorialPublicSubnet
      Tags:
        - Key: "Name"
          Value: "NAT Gateway"

We use the NatGateway resource to create our new NAT gateway and indicate that we need our VPCGatewayAttachment resource provisioned before this is created. We set the SubnetId property to put it in our public subnet so that it can communicate with the internet.

While it would make a lot of sense to provide the Elastic IP we provisioned directly to the gateway, we instead need to provide the AllocationId associated with that Elastic IP instead. Here we use the !Sub function to get the AllocationId of our NatGatewayElasticIP and provide that value to the AllocationId property.

Now we can setup our route for the private subnet to get to the internet via our NAT gateway.

  TutorialPrivateSubnetNatRoute:
    Type: AWS::EC2::Route
    DependsOn: TutorialNatGateway
    Properties:
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref TutorialNatGateway
      RouteTableId: !Ref TutorialPrivateRouteTable

We create our route, using the DependsOn attribute to indicate that we need the NatGateway provisioned before this route is created. Then we use the NatGateway property to link in our gateway and the RouteTableId to tie the route to our routing table.

  TutorialPublicSubnetRouteAssoc:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref TutorialPublicRouteTable
      SubnetId: !Ref TutorialPublicSubnet

  TutorialPrivateSubnetRouteAssoc:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref TutorialPrivateRouteTable
      SubnetId: !Ref TutorialPrivateSubnet

The final step is to associate our routing table with our subnets! We use the SubnetRouteTableAssociation to attach our table of routes to the matching subnet.

Setup Our Security Groups

Before we can provision our instances we need to get our security groups set. We will have one for each subnet and these groups will specify what traffic we will allow in and out of each of our subnets. We will allow web traffic in and out of our public subnets, in our private subnet we’re only going to allow web traffic out and nothing inbound.

  TutorialPublicSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Description: "Web Server Security Group"
    Properties:
      VpcId: !Ref TutorialVPC
      GroupDescription: Web Server Security Group
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: 80              # HTTP 
          ToPort: 80
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: 443             # HTTPS
          ToPort: 443
        - CidrIp: 0.0.0.0/0         # Replace with your IP Range
          IpProtocol: tcp
          FromPort: 22              # SSH
          ToPort: 22
        - CidrIp: 10.6.0.0/24       # Inside the VPC
          IpProtocol: -1            # All traffic

We use the SecurityGroup resource to create our new group for the public subnet. The Description we provide is visible in the web console and then we move on to the properties. We use the VpcId property to indicate that this group is for our VPC and then provide another, very similar description, for the GroupDescription (this property is required).

With that out of the way, we move on to the in-bound (ingress) rules. Each rule is provided by a Security Group Rule property that is made up of an IP address range, the protocol and then a range of ports. We have created rules for HTTP, HTTPS, and SSH and traffic inside of our VPC. If you were provisioning a Windows instance, don’t forget to add port 3389 for Remote Desktop access.

In the example above we have opened up the SSH port to the entire internet: this is not a great idea. Since these ports are used almost exclusively for management of the instances, a better choice would be to open these ports up to the address range that is used by your office. If you don’t have a fixed range of addresses you can use then I encourage you to look into other solutions, perhaps using a VPN.

      SecurityGroupEgress:
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: 80              # HTTP
          ToPort: 80
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: 443             # HTTPS
          ToPort: 443
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: 587             # SSL SMTP
          ToPort: 587
        - CidrIp: 10.6.0.0/24       # Inside the VPC
          IpProtocol: -1            # All traffic
      Tags:
        - Key: "Name"
          Value: "Public Subnet"

Here we setup the rules for outbound traffic: HTTP, HTTPS, and SMTP over SSL. We also have a rule that allows all traffic inside our VPC. If you are more security conscious, you can nail down specific types of traffic inside your VPC (i.e., just web, SSH and database traffic).

The security group for our private subnet is much simpler.

  TutorialPrivateSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Description: "Private Security Group"
    Properties:
      VpcId: !Ref TutorialVPC
      GroupDescription: Private Subnet Security Group
      SecurityGroupIngress:
        - CidrIp: 10.6.0.0/24       # Inside the VPC
          IpProtocol: -1            # All traffic
      SecurityGroupEgress:
        - CidrIp: 10.6.0.0/24       # Inside the VPC
          IpProtocol: -1            # All traffic
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: '80'            # HTTP
          ToPort: '80'
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: '443'           # HTTPS
          ToPort: '443'
      Tags:
        - Key: "Name"
          Value: "Private Subnet"

We are only allowing incoming traffic from inside our VPC into the private subnet. Outbound, we allow all traffic destined for other addresses in our VPC and we’re allowing only HTTP and HTTPS to the public internet. Traffic from addresses in our private subnet is already being routed through our NAT gateway, we’re likely only going to use the internet to download software updates.

Provision Other Resources

There are a lot of other resource that we might provision, aside from instances. You might need queues, notification topics, etc. For this project we’re going to setup an S3 bucket to hold dump files from our database server.

The nice thing about setting these up before the instances is that we can tell the instances about these resources and set policies so that our instances can access these resources without providing them with a specific set of credentials.

  TutorialBackupS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private
      Tags:
        - Key: "Name"
          Value: "Backup Bucket"

Simple enough! We use the Bucket resource to create our bucket and we set that bucket to be private with the AccessControl property.

It was a long walk but we have accomplished a lot. Go ahead and provision your template and wait for CloudFormation to finish getting everything setup. From the CloudFormation web console, select your stack and click on the “Resources” tab to see all of the items that you have provisioned. Such progress!

Until Next Time

That is all we are going to cover in this article, we went over a lot of material. In the next piece in the series we’ll setup some groups and roles and actually provision some instances.

Thank you for reading!

Linux Is All About Choices

2014-07-10T00:00:00-04:00

Downloading the latest Ubuntu release and installing it on your laptop or in a virtual machine is easy, but maybe it’s too easy. It’s too easy to blindly accept the choices that the distribution has made for us. Maybe it’s also too hard, there’s little room left for us to customize the environment or mold it to our own preferences and habits.

It’s Too Easy
Maybe We Don’t Need the Magic
What Choices Didn’t We Make?
- How Does This Thing Really Work?
- All That Fancy GUI Stuff
Challenge Yourself
What Do I Use?

It’s Too Easy

I understand that there are people who come from another environment, like Microsoft’s Windows or Apple’s OSX, who want something that doesn’t challenge them. They want to send an e-mail or write a paper or add some numbers and the rest is kind of like icing on the cake. I can see how a distribution like Ubuntu would appeal to them and I agree that it’s a good thing.

But what about those of us who spend most of our workday (and some of our non-workday) in front of a computer? For many of us an all-in-one distribution is too easy, we forget that we can make the environment our own.

Maybe We Don’t Need the Magic

Friends and family who use Apple products love them and enjoy talking about them, once of the things I hear about most often is “the magic.” How it knows how to do the right thing when they plug in their backup drive or dock their phone. How a new phone contact magically turns into a new e-mail contact. For sure this stuff is great, but what if you want to do it a little different? That’s when the magic stops being so magical and starts to become frustrating.

Over on the Linux side we don’t need to be so magical. It’s really nice when the computer does the thing you want all on it’s own, as if it can read your mind. But unlike Apple there’s no need to hide the mechanisms that make the magic happen. Most of the time these functions can be easily customized if we happen to know what does what when. Unfortunately many distributions are working hard to hide these mechanisms and make them hard to manipulate. I think Gnome Shell is a good example of this; people new to Linux love it while those of us who are more demanding are frustrated more and more with every release.

What Choices Didn’t We Make?

Installing a distribution like Ubuntu is super easy, you boot from the image and the graphical installer asks you where you live, what time it is, and you pick a name and password. It’s painless, for sure, but it’s also letting the (well intentioned) folks who assembled that distribution make a lot of decisions on our behalf. Decisions that we often didn’t even know where choices at all.

How Does This Thing Really Work?

I’ve been using Linux for a good long while now and there were some frustrating things that I simply took for granted because I didn’t understand how they really worked. For sure, most of that was laziness but the distribution was at least partly to blame for hiding all of the machinery. Hiding it, I suppose, so that I wouldn’t grow confused by all of the choices.

I have a docking station at work with two displays and another at home with just the one display. For many moons I would grumble when docking or un-docking and re-arranging the displays. It wasn’t until pretty recently that I figured out I could write a simple udev rule that triggers every time the laptop plugs into or out of a dock. Combine that with autorandr configurations for the displays and now I don’t have to fuss with these settings at all. It just works and most importantly: not by magic!

This is an example from my own experience and I have more. The same mechanism let’s you write a rule that, maybe, kicks off a backup when you plug in a particular USB drive. Distribution packagers aren’t trying to hide anything but it’s often the case that you just don’t know what you don’t know.

All That Fancy GUI Stuff

Probably the choices we most miss out on are the ones that involve our desktop environment. Many distributions have already chosen the entire desktop stack for us and it’s often difficult to move away to something else. There are several Ubuntu flavors that are bundled with different desktop environments but they work much the same way. Part of the problem is the way these components overlap and another is that it’s a big change and no one wants to “mess it up.” But making it difficult to switch from one desktop environment to another isn’t helping anyone.

I’m a strong believer in the tiling window manager and there are a lot of them out there, many of them are easy to customize for your own work environment. Ubunutu’s Unity environment is great (or tolerable), to be sure, but it’s not so easy to change the way it works. Instead of molding the environment to fit ourselves, more often than not we adjust the way we work instead.

I’ve been using the i3 window manager for over a year now and I really like it. The things that maybe didn’t work exactly right for me I went ahead and changed so that they did. I wanted to change the items in the status bar and it was so straightforward that I ended up writing my own.

And there are more options out there. Awesome let’s you customize the window manager with Lua, xmonad let’s you customize it’s behavior with Haskell. And so on, the point being that not every tool is buying into this idea that letting people customize their environment is somehow a bad idea.

Challenge Yourself

I think we would all do better to challenge ourselves, to make things a little harder. Instead of choosing a Linux distribution on the basis of how easy it is to install, instead pick one that lets you make choices at every step. Choices like….

Do you want to encrypt any of your disk partitions?
Which bootloader would you like to use?
Which display driver do you really want to use?
Which display manager would you like to use?
Would you like a desktop environment or just a window manager?

Yes, I’m recommending Archlinux. In my humble opinion, you’ll learn a much more about how everything works together just by spending a little time on these questions. They are all worth asking and many of them will impact you every time you sit down with your machine. Making these kinds of choices might seem like minutia at first they build on one another. At the end of the day, you might be surprised how good you can have it.

What Do I Use?

On the off change that you’re curious, this is what I use daily on my laptop. Sure, it’s a little ugly but it’s mine.

ArchLinux is my distribution
Gnome Display Manager, because pretty
i3 Window Manager, I like keeping my hands on the keyboard
i3 dmenu desktop for launching applications
Compton for window compositing
Dunst for notifications
My Own Custom i3 Status Bar
Autorandr to keep my displays sorted
Guake and Termite for the command line
fish, the friendly interactive shell
dwb is my primary web browser
Emacs and IntelliJ IDEA for editing code
Notmuch for mail management with the Emacs client
Ranger for managing my files

Real Life Web App Integration Testing (IT) with Spring

2013-03-30T00:00:00-04:00

I know what you’re thinking: “what constitutes ‘real life’ integration testing (IT)?” In my opinion, “real life” means “complicated web application” or maybe “web app that is stuck to creaky database not entirely under my control” or “web app that uses auditors to enforce security” or, probably most inclusively “web app that leverages a lot of Spring”.

I'm sure you know what I mean: the kind of application they pay you to work on and, clearly, not something you designed from scratch to be relatively clean and modular. Not your own pet project.

Doesn’t Spring Have Awesome IT Support Already?
Our Test Project
The POM.XML File
The Test
The Idealized Case: You Just Saw It and Your App Isn’t It
Your Mileage May Vary

Doesn’t Spring Have Awesome IT Support Already?

Spring does have some good IT tools in place already, things like the TestContext framework and some special transaction management that will roll back any database changes that your test may create. It works pretty well but not for all cases; it really works best when you’ve planned for it ahead of time and often not so well if you’re only thinking about it after a good size of the code base is in place.

I think there are some positive points to the Spring approach but I’d argue that when your doing your integration testing, it’s kind of an oxymoron to test your components in isolation.

You’ll also run into problems if your doing in-browser testing and your application is at all complicated. With your server running in one context and your test in another, the test transaction manager is going to frustrate you by keeping your changes invisible to your server. If you want fixture data, you can use something like DbUnit, but if your test is too complex it can be problematic to keep everyone’s test data in sync. If your team is working with one central test database this isn’t a viable solution at all.

These are the issues I’m going to attempt to address. I think there are some positive points to the Spring approach but I’d argue that when your doing your integration testing, it’s kind of an oxymoron to test your components in isolation.

Our Test Project

Our test project will be simple but we’ll pull in a good portion of Spring’s features. We’ll use Spring Data with JPA for persistence and Apache Wicket to provide our web application. The application itself is pretty simple, it provides a simple to-do list and it lets you add "contexts" (the GTD-ism) as well as add and remove to-do items. You can take a look at the test project over on GitHub.

We’ll use Selenium with Ghostdriver and PhantomJS for our integration testing. Selenium will provide a nice Java API for managing the web browser and PhantomJS gives us a fast, headless browser. In my opinion, if you would also like to test real browsers you’re better off making your continuous integration environment perform those tests. You don’t want your developers bogged down by a lengthy and time consuming integration test phase; PhantomJS lets them run the IT tests as fast as possible.

The POM.XML File

As you might have guessed there’s a lot of good stuff setup in this project’s POM file. We’ll go over the parts relevant to IT testing to make sure you don’t miss anything.

Dependencies

We need to add JUnit, Selenium and Ghostdriver dependencies to our project. This is pretty straightforward.

<dependencies>
  ...
  <!-- Testing -->
  <dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.10</version>
  </dependency>
  <dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>2.31.0</version>
  </dependency>
  <dependency>
    <groupId>com.github.detro.ghostdriver</groupId>
    <artifactId>phantomjsdriver</artifactId>
    <version>1.0.3-dev</version>
</dependencies>

We pull in JUnit, Selenium and Ghostdriver (the PhantomJS driver for Selenium). You’ll need to have PhantomJS installed already, as long as it’s in your current path you won’t need to do any configuration at all.

A Note About PhantomJS

As of this writing, there is a known bug in the version of Ghostdriver bundled with PhantomJS 1.9.0; it has some issues switching over to an Iframe of content. Our test project uses Iframes for modal pop-up windows so you’ll want to build your own PhantomJS with the newest version of Ghostdriver, it’s all clearly documented on the Ghostdriver project page. If you’re on 64-bit Windows you can also use the one that I compiled.

The Jetty Plugin

When we run our integration tests we need Jetty to startup and load in our application. It needs to stay running while our tests run and it needs to cleanly shutdown when our tests are over. The Jetty Maven Plugin makes it easy to do just this. First, we set some properties up to let Jetty know what ports to use.

<properties>
  <jetty.stopKey>stopJetty</jetty.stopKey>
  <jetty.stopPort>8081</jetty.stopPort>
  <jetty.port>8080</jetty.port>
 </properties>

Here we tell Jetty what port to listen to for stop requests and which port it should use to provide our application.

Next we setup the plugin itself.

<build>
  ...
  <plugin>
    <groupId>org.mortbay.jetty</groupId>
     <artifactId>jetty-maven-plugin</artifactId>
     <version>8.1.8.v20121106</version>
     <configuration>
       <webApp>
         <contextPath>/</contextPath>
         <descriptor>src/main/webapp/web.xml</descriptor>
         <resourceBases>
           <dir>src/main/webapp</dir>
           <dir>src/test/webapp</dir>
         </resourceBases>
      </webApp>
    </configuration>

In the stanza above, we add the plugin to our project and tell it where to find our web application…

    <executions>
      <execution>
        <id>start-jetty</id>
        <phase>pre-integration-test</phase>
        <goals>
          <goal>run</goal>
        </goals>
        <configuration>
          <daemon>true</daemon>
          <stopKey>${jetty.stopKey}</stopKey>
          <stopPort>${jetty.stopPort}</stopPort>
        </configuration>
      </execution>
      <execution>
        <id>stop-jetty</id>
        <phase>post-integration-test</phase>
        <goals>
          <goal>stop</goal>
        </goals>
        <configuration>
          <stopKey>${jetty.stopKey}</stopKey>
          <stopPort>${jetty.stopPort}</stopPort>
        </configuration>
      </execution>
    </executions>
  </plugin>
</build>

Then we set Jetty to run right before our integration tests and to stop when the tests are complete. In this setup, Jetty will run in “daemon” mode so that Maven can move on and run our tests.

Tell Surefire to Skip Our IT Tests

We don’t want the regular Surefire plugin to run our tests during the regular “test” phase, we need to explicitly tell Surefire to skip them. This also goes in the “build” section of our POM.xml file.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <excludes>
      <exclude>**/integration/**/*.*</exclude>
    </excludes>
  </configuration>
</plugin>

Tell Failsafe to Run Our IT Tests

The Failsafe plugin will actually run our tests. This plugin is very similar to Surefire, it justs runs the test at “integration test” time (the “verify” phase”) instead of during the “test” phase.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>2.12.4</version>
  <executions>
      <execution>
          <id>integration-tests</id>
          <goals>
              <goal>integration-test</goal>
          </goals>
          <phase>integration-test</phase>
          <configuration>
              <includes>
                  <include>**/integration/**/*.*</include>
              </includes>
          </configuration>
      </execution>
      <execution>
          <id>verify</id>
          <goals>
              <goal>verify</goal>
          </goals>
      </execution>
  </executions>
</plugin>

There’s not too much happening here, we setup a new “goal” called “integration-test” and we set this goal to target the “verify” phase. If any of our IT tests fail, Failsafe will ensure that the entire build fails as well.

Phew! That’s a lot of stuff.

The Test

With all of the infrastructure in place, we’re ready to write up our first IT test. I’ll go through the important bits next, you can look at the whole test over here.

Instantiate a New PhantomJS Before Each Test

Creating a new PhantomJS instance is pretty cheap so we’ll create a new one before each test is run.

private WebDriver driver = null;

@Before
public void setUp() {

  DesiredCapabilities desiredCapabilities = new DesiredCapabilities();
  driver = new PhantomJSDriver(desiredCapabilities);
  driver.manage().window().setSize(new Dimension(800, 600));
  driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
}

Note that we set the size of the browser window, if you don’t explicitly set this it will be far too small.

Also interesting is how Selenium finds elements on the page. Whenever you tell it to find something on the page, it will scan through the DOM of the page until it finds the element you want and it will then return the found element or “null”. If you tell Selenium to “implicitly wait”, it will scan the page continuously for the element until it either finds it or the wait time is exceeded. This is a great fit for pages that have dynamic content, Selenium will scan the page until the element appears.

The Simple Case: Ensure the Page Renders

Here’s a very simple test that loads in a page and checks for a couple items of content.

@Test
public void testHomePageRenders() {

  driver.get("http://localhost:8080/");

  Assert.assertNotSame(driver.findElements(By.id("title")).size(), 0);
  Assert.assertNotSame(driver.findElements(By.id("addTodoLink")).size(),
    0);
  Assert.assertNotSame(driver.findElements(By.id("addContextLink"))
    .size(), 0);
}

Pretty simple! Selenium loads the page and then searches for some elements. We then check to see that the number of returned elements is not zero.

The Tricky Case: Fill Out a Modal With a Form

Now the tricky one, we want Selenium to load the home page and click on the button that creates a new “context”. When the modal pop-up window appears we’d like the form filled out with some test data and then have Selenium click on the form’s “save” button. Lastly we’d like Selenium to make sure that the new context appears on the home page. These are all AJAX requests, our to-do application is a single page application.

@Test
public void textAddContext() {

  driver.get("http://localhost:8080/");

  // get all of the current context identifiers
  final List<String> contextIdList = new ArrayList<String>();
  for(WebElement context : driver.findElements(
    By.cssSelector(".context"))) {
      contextIdList.add(context.getAttribute("id"));
  }

  // open the "add context" modal
  driver.findElement(By.id("addContextLink")).click();
  Assert.assertNotSame(driver.findElements(
    By.className("wicket_modal")).size(), 0);

  // switch focus to the modal
  WebElement webElementModal = driver.findElement(
    By.className("wicket_modal"));
  logger.info("Modal: " + webElementModal + ", " +
    webElementModal.getAttribute("id"));
  driver.switchTo().frame(webElementModal);
  logger.info("Focused on modal window");

  // verify form elements rendered, fill out the form
  String contextName = "Test Context (" + (new Date().getTime()) + ")";
  Assert.assertNotSame(driver.findElements(By.id("contextAddNameField"))
    .size(), 0);
  driver.findElement(By.id("contextAddNameField")).sendKeys("Test Context");
  Assert.assertNotSame(driver.findElements(
    By.id("contextAddDescriptionField")).size(), 0);
  driver.findElement(By.id("contextAddDescriptionField"))
    .sendKeys("For all the testing");
  Assert.assertNotSame(driver.findElements(
    By.id("formCancelLink")).size(), 0);
  Assert.assertNotSame(driver.findElements(
    By.id("formSubmitLink")).size(), 0);

  // submit the form
  driver.findElement(By.id("formSubmitLink")).click();

  // switch focus back to the main window
  driver.switchTo().defaultContent();

  // wait for the new context to appear
  WebElement webElementContextNew =
          (new WebDriverWait(driver, 30)).until(
             new ExpectedCondition<WebElement>() {

      @Override
      public WebElement apply(WebDriver webDriver) {

          // holder for the new context element
          WebElement webElementNew = null;

          // loop through all of our context identifiers
          for(WebElement context : webDriver.findElements(
            By.cssSelector(".context"))) {

              if(!contextIdList.contains(context.getAttribute("id"))) {
                  logger.info("New context found with ID " +
                    context.getAttribute("id"));
                  webElementNew = context;
              }
          }

          return  webElementNew;
      }
  });
  Assert.assertNotNull(webElementContextNew);
}

You can see that there’s a lot more going on here. In order to make this test more robust, we’re making a list of the current contexts before we add our new one. After we submit our form, we know our new context has appeared on the page when the list of contexts grows by one element.

The nice thing about Selenium is that the code is pretty easy to follow, it’s just more Java code. In my opinion the trickiest part is figuring out which parts of the page you need to wait for and then writing the right code to test the change.

The Idealized Case: You Just Saw It and Your App Isn’t It

It’s important to note that the example above covers a pretty idealized case, the same sort of application that most of the Spring with IT documentation spends its time on. It’s all pretty straightforward and it all works really great. Still, before we move on lets recap the assumptions we’ve made so far:

We don’t need any fixture data and our tests will interact only with the web application. There’s no access to our Spring service or data layers at all.
We’re using an ephemeral data store and we’re not worried about cluttering it up with stuff. In the above example we didn’t talk about it but it’d be reasonable to use an in-memory database.

Idealism, like it or not, is a exclusionary. By assuming a brand new application and presuming minimum complexity we’ve unwittingly excluded most real-life applications; adapting the code above to your own existing project may not work out so well. Such is life.

Access to the Spring Container

It doesn’t need to be this way, let’s address the first item: we need access to the Spring container in order to create and delete fixture data for our tests.

The first thing we need to do is to start using the SpringJunit4ClassRunner, this will instantiate a Spring container and then run our IT suite in the context of this container. Simply adding an annotation to the top of the test class and then providing a ContextConfiguration that points to our Spring configuration will be enough to get this working.

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"classpath:/applicationContext.xml"})
public class ToDoHomePageTest {

  // test code goes here...
}

If you try to do this with our test application, you’ll see that this is a little more complicated than it looks. In our case we’re using an embedded Derby instance in our web application and when we try to spin up a second Spring configuration it fails because the files that Derby is using are locked. This isn’t a failure on the part of Spring, it’s just the way it is: you probably don't want to use an embedded database when your doing this kind of testing. You could fudge it by having the test Spring instance connect to the running Derby instance but we’re not going to go over that here, this article is already way too long as it is.

Transactions That Persist

Let’s assume that we’ve moved to a standalone database that already contains our test data, the next problem we’ll encounter is that no matter how we annotate our tests, the data never makes it to the database. This is a feature of the Spring JUnit runner: the transaction manager provided rolls back the transaction once the test completes. This is a nice idea but in the context of IT tests, not super useful. After all, we need our fixture data to make it into the database so that our web application can see this data and our tests can interact with it.

The fix here is easy, just ask Spring for a real transaction manager and use the TransactionTemplate to get a real-life transaction for your operation. Also, use the TransactionConfiguration annotation to set the “defaultRollback” property to false.

@TransactionConfiguration(transactionManager = "txMgr",
  defaultRollback = false)
public class ToDoHomePageTest {

  @Autowired
  private PlatformTransactionManager platformTransactionManager;

  @Autowired
  private ContextService contextService;

  private TransactionTemplate transactionTemplate;
  private Context context = null;

  @Before
  public void setUp() {

    transactionTemplate = new TransactionTemplate(
      platformTransactionManager);

    context = transactionTemplate.execute(
      new TransactionCallback<Context>() {

      @Override
      public Context doInTransaction(
        TransactionStatus transactionStatus) {

        Context context = new Context();
        context.setName("Test Context");
        context.setDescription("For all the testing.");

        return contextService.save(context);
      }
    }
  }

  @Test
  public void testAddToDo() {

    // test code here
  }

  @After
  public void tearDown() {

    transactionTemplate.execute(new TransactionCallback() {
      return contextService.delete(context);
    });
  }
}

We’ve added an AutoWired field for the transaction manager and the context service. With these in hand we can go ahead and create our fixture data, run our test and then delete the data when the test are complete. As an added bonus, this will also let you test with Hibernate's Auditors.

Where Are My Properties?

So far so good but if you’re loading data from *.properties files you’ll notice that those files aren’t picked up by the test runner. Lucky for you there’s another annotation for that, it may be placed above your class declaration.

@PropertySource({"classpath:spring.properties",
  "classpath:other.properties"})

This will read in those two files of properties and make those properties available to your test Spring configuration.

Two Incompatible Beans?

The last gotcha' I've seen appears when your test Spring configuration is starting up, it complains that there is more than one bean available but that they are incompatible and thus the field can't be autowired. This one took me a while to track down, in my case there were mock service beans being instantiated to support regular unit tests and because Failsafe pulls all of the test classes into the classpath, these were clashing when the IT tests ran. The solution is simple: add an exclusion for these test classes in your Spring configuration file.

<context:component-scan base-package="com.windsorsolutions.todos">
  <context:exclude-filter type="regex"
     expression=".*ContextUnitTestConfiguration"/>
</context:component-scan>

The regular expression above excluded these mock beans and everything started working.

Your Mileage May Vary

My hope is that this article covers most of the issues you may run into when running IT tests against your Spring project. Still, Spring provides a lot of stuff and there may be issues lurking out there. If you run into anything really interesting, let me know and I'll add it to this article!

Bishop: Makes Your Web Service Shiny

2012-05-07T00:00:00-04:00

A couple of months ago I was working with another developer, he was building a web application around a web service that I had just finished writing. We were going over the API documentation that I put together and he straight-out asked me: “What happens if I send it something that’s not JSON?” I answered honestly: “I don’t know, but I bet it logs a really big stack trace!”

“What happens if I send it something that’s not JSON?”

“I don’t know, but I bet it logs a really big stack trace!”

This sort of thing (authoring services) is becoming a bigger and bigger part of my work and it’s inexcusable that I don’t know more about how HTTP actually works. In this case the application was implemented in Clojure and used Ring to handle the web server part, anything that validated the content being sent to the application I had to write myself. And I did, kind of; if the JSON parser couldn’t parse the data the client sent then it logged a stack trace.

Really, I should check the incoming “Content-Type” header that the client provided and if wasn’t JSON I shouldn’t even try to parse the data. That’s not so hard, but did I want to be re-writing that code for every project? And what about all of the other HTTP headers? To be honest, I didn’t even know what most of them were supposed to be doing.

Naturally this is a problem that’s been solved before. After doing some research I found Webmachine, which goes a long way towards handling all of the HTTP level stuff. Unfortunately for me, it’s written in Erlang and that means writing your service in Erlang; I really need to work with Java and Clojure. Still, not one to give up lightly, I began writing my own Webmachine-like library in Clojure and thus Bishop was born.

More Than a Pipe Into Which You Shovel Data
What Does Bishop Do?
How Do I Get Started with Bishop?
Conclusion and Wrap-Up

More Than a Pipe Into Which You Shovel Data

After taking a closer look at HTTP and it’s specification, it was clear that it could do a lot more than I had thought. Looking back on past projects, it’s painfully obvious that I’ve been taking what is really an application protocol and ignoring all of the interesting bits, instead using it as little more than a pipe to push documents through. I’ve been using either the requested URL or parameters or maybe even neither and simply examined the body content, thus eliminating any of the real advantages of using HTTP in the first place.

And there are advantages. The protocol is already thinking about caching your data where it makes the most sense. There’s already an algorithm for taking the list of content types that the client wants and the content types the server provides and picking the best match. It can manage safe updating of resources as well as notifying the client of conflicts. And so on, by ignoring what the HTTP protocol has to offer I was making more work for myself.

What Does Bishop Do?

Like Webmachine, the Bishop library handles all of the HTTP protocol level stuff, you won’t have to spend a lot of time thinking about it. It does this by letting you code up a set of callback functions that the library will invoke as needed. For instance, when you’re using Bishop you can provide a function that lists the content types that your application knows about (i.e. “text/html”, “application/json”, etc.) and another that lists the request methods your application can handle (i.e. “GET”, “PUT”, etc.) Bishop will invoke the functions at the right time in order to negotiate with the client. When everything is all figured out (what content type to provide, if the URL points to a valid resource, etc.), your handler function is invoked and you can do whatever it is that your application does.

The idea is to provide a relatively small library that will make your life easier and hopefully more pleasant by making it straightforward to provide a consistent web service API that obeys HTTP semantics. It will make the lives of those around you easier as well, clients can expect your service to respond to the common HTTP request methods with reasonable responses. Placing caches around your service will also be much simpler and you’ll have some level of control over how your service’s data is cached.

How Do I Get Started with Bishop?

The Bishop library was designed to be small and easy to use. The basic idea is that you will code up a set of Bishop resources, each resource will correspond to a particular URL path. These resources will provide the necessary functions that Bishop needs in order to negotiate the request with the client and decide which handler function to invoke. Once that step is complete, your handler function will likely invoke a function or set of functions that provide the core functionality of your application; perhaps calling functions that add a record to a database or fetch data from your application.

We’ll wall through the sample application that accompanies the library and explain how the pieces fit together.

Add the Dependency

If you’re using Leiningen to manage your project, getting started is as easy as adding a dependency to your “project.clj” file.

(defproject bishop-sample "1.0"
  :description "A sample Bishop application"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [ring/ring-core "1.0.2"]
                 [ring/ring-jetty-adapter "1.0.2"]
                 [cheshire "4.0.0"]
                 [hiccup "1.0.0"]
                 [tnrglobal/bishop "1.0.2"]]
  :dev-dependencies [[swank-clojure/swank-clojure "1.3.3"]]
  :main com.tnrglobal.bishopsample.core)

In the example above, we’re using Ring and Jetty to provide the web service, Cheshire to handle JSON parsing and Hiccup to provide HTML output. When you build your project, Leiningen will fetch all of these dependencies and anything they might depend on.

The Application

Next we need the application that we’ll be wrapping with our service interface. In this example we’ll be providing a simple to-do list with functions for listing all of the to-dos as well as adding, removing and updating items.

Each to-do item will be represented by a hash-map containing a title, description, the date it was created and the date of it’s last modification. First we setup a reference to hold our “database” of to-do items and we’ll start it off with one entry. I won’t go over this stuff function-by-function, you can take a look at the implementation here.

The Service

Each Bishop resource is defined with two hash-maps. The first provides a map of content types provided by the resource and the functions that accept a Ring request map and return data in that format. The second hash-map contains the key for a callback function and the actual function, these are invoked by Bishop when processing each request. You can view the complete list of callback functions here.

We can now work on developing the Bishop resources that will provide our web service. We’ll be using the “/todos” URL to provide both a list of to-do items (with a “GET”) and an end-point for requests to create new items (via “POST” or “PUT”). While this application deals with JSON data we also want to present a simple HTML interface for viewing the data. Listed below is our resource definition.

(def todo-group-resource
  (bishop/resource

   {;; JSON handler
    "application/json"
    (fn [request]
      (cond

               ;; return a sequence of to-do items
               (= :get (:request-method request))
               {:body (generate-string
                        (map (partial add-resource-links URI-BASE)
                             (app/todos)))}

               ;; create the new to-do item
               (= :put (:request-method request))
               (let [todo-in (parse-string (slurp (:body request)) true)
                     todo (app/todo-add todo-in)]
                 {:headers {"Location" (str URI-BASE "/" (:id todo))}})))

    ;; HTML handler
    "text/html"
    (fn [request]
      {:body (xhtml {:lange "en" }
                    [:body
                     [:h1 "To-Do List"]
                     [:ul (map #(todo-short-html
                                  (add-resource-links URI-BASE %))
                               (app/todos))]])})}

   {;; the request methods supported by this resource
    :allowed-methods (fn [request] [:get :head :post :put])

    ;; POSTs with new data will be handled like a PUT
    :post-is-create? (fn [request] true)

    ;; we use the modification date on the most recently modified
    ;; to-do item as the modification date for the list of items
    :last-modified (fn [request]
                     (app/most-recent-todo-modified))}))

The first map defines the content types our resource accepts and the functions that handle each type of request. Our first handler accepts JSON data and can handle either a “GET” or a “PUT” request. You can see that when the client fetches data we add resource links to each item and then use the resulting hash-map to create our JSON output. This output is then placed under the “:body” key of the response object. It the client is providing new to-do item data with a “PUT” request, we’ll parse that JSON data into a hash-map and add that item to our database. We’ll then redirect the client to the URL for the new to-do item.

Our second entry handles HTML content. When the client makes a “GET” request for HTML to our resource, we’ll provide an HTML listing of the items (including a link to the resource for each item). We don’t have to worry about parsing out the client’s headers and figuring out which type of content the client wants. Bishop handles that all of that for us.

We define our callback functions for our resource in the second hash-map, we provide the information that Bishop needs to handle incoming requests. We specify the methods that our resource can handle and instruct it to treat “POST” requests as if the client was doing a “PUT” for a new item. We also tell Bishop how to correctly generate the “Last-Modified” header.

There’s much more that Bishop can do for you, take a look at the rest of the code for this service. For instance, when you tell Bishop how to generate an “ETag” for your resource you make it easy for clients to fetch the entire resource only when it actually changes. If you run this sample application and point Google Chrome at it, you’ll notice that the first time you view a to-do item in the browser it fetches the item from the server. Subsequent fetches are served from Chrome’s local cache: Chrome uses the ETag header to do a conditional fetch and as long as the resource isn’t updated, Bishop responds with a “304 Not Modified.”

Setting up Routes and Starting the Server

Lastly, we need to set our application up with Ring and a server, in this example we use Jetty. First, our routes.

(def routes
  {["todos"] todo-group-resource
   ["todos" :id] todo-item-resource})

We’re using Bishop’s router, we indicate that everything with a “todos” URL should be directed at our group resource and anything in the form of “todo/???” should hit our item resource. The router provided is modeled after Webmachine, if you prefer another router you could simply use that instead.

Next we setup our Ring application. No magic here, we provide our routes (which point to Bishop resources) to our Bishop handler. We can then wrap that handler in any required middle-ware (none in this case) and hand that off to Jetty.

;; we setup our ring application
(run-jetty (bishop/handler routes) {:port 3000})

Conclusion and Wrap-Up

The Bishop library provides the tools needed to easily create a web service that understands the semantics of HTTP. While this is something that you could always do on your own, the library makes it easy by encapsulating all of the logic required to understand the client’s request headers and then directing it to the correct handler on the application side. Under the covers Bishop implements finite state machine that models the HTTP protocol, ensuring that requests are handled the same way every time.

It is early days for this project, this is only the first release and I have no doubt that there are bugs to be uncovered! If you happen to spot any, please take the time to log an issue. Help of any kind is greatly appreciated and we’re looking forward to accepting pull requests.

Why Is My Web Service API Crappy?

2012-04-02T00:00:00-04:00

I knew that I hard written a crappy web service API but why was it so crappy? Was it because, no matter what, it only returned JSON data? Was it because the arguments were passed via their positions in the URL? Was it because the version of the API was part of the URL? Was it all of the above and, if so, how was it supposed to be done?

I’ve been doing a lot of research and reading and I think I have a better idea of how this all should work. For the sake of getting all of my ideas down in one place, I’m going to crudely regurgitate my current thinking in this article.

Why It’s Crappy
Another Way
A Really Simple Web Service
Caching
Modeling a Protocol
You Don’t Need to Do it All On Your Own

Why It’s Crappy

Right off the bat, I knew my API was weak in that it completely ignored the HTTP protocol. If you’re writing a service that speaks HTTP (and I was), you should probably know a little about how HTTP works (and I didn’t). This is the sort of thing those crazy REST people have been crowing about forever and now I can understand why: there’s a lot more meat in there than I had thought. For sure, it won’t solve all of your problems but it can solve some of them.

Another Way

If I made my API work a little bit more like how web servers already work, I could make things easier on everyone. I’ve always worked with a pretty lightweight framework to put these APIs together (i.e. CherryPy or Ring) and while this works okay, it left lot of the work of handling the various HTTP headers up to the me. I mostly skipped all of that stuff because I needed to get things done. If you’ve done the same, trust me, there’s no judgment here. But there are some good reasons to implement more of this stuff.

Give the Client What the Client Wants

For one thing, the client can tell you very specifically what kind of content it wants. Maybe it wants XML, if you really only serve JSON data you can go ahead and say so. Maybe the client does want JSON data but only an older version of your API. That’s right, you don’t need to put the version in the URL, finessing the requested content type can serve the same purpose.

Tell the Client Where To Go

I usually end up publishing more and more URLs as more functionality is needed. It all ends up looking very ad-hoc because it’s all very ad-hoc. But what if we provided pointers to the client instructing it where it should go to get the information that it wants or to change the state of a resource? We’d end up with much looser URLs and we’d be able to change them when we need to. While the client needs to do a little more work to parse out these URLs, I think it’s definitely worth it.

Provide Helpful Hints

The client can submit the “OPTIONS” request to find out what actions (i.e. “GET”, “POST”, “PUT”, etc.) are valid for the URL. Sure, you can put that in the documentation but the client can also discover it on their own. Much of the time the correct method will already be assumed; i.e. you POST to create a new resource and PUT to update an existing item.

URLs Can Represent State Changes

Once we get our API to the point where the client is following the links that we provided with our responses, we’re in a place where the state of a particular item changes as the URLs our API provides are followed. If a particular action isn’t valid for a particular item we simply do not provide a URL for that action.

This isn’t earth-shattering news, it’s just not something I hadn’t ever thought about before. But now that I’m thinking about it, it’s clearly a much better way to manage these state transitions.

Predictable Caching

Caching the results of your API can be a real pain point, especially if you’re not providing the information that the caching layer really needs to be effective. For the longest time I didn’t have to worry about putting a cache in front of my web service, they were all internal. Once I did, it was made clear to me how difficult my service was making it for everyone else.

A Really Simple Web Service

Okay, enough talk. What does an API that leverages these ideas actual look like? We’ll use a simple example, a web service that exposes a single to-do list. First, we want to get a list of all of the items on our list, this represents the main entry point to our API. I’m not going to list all the headers, just the relevant ones.

GET /todos HTTP/1.1
Host: somesite.com
Accept: application/vnd.com.somesite.todos-v1+xml

Date: Mon, 02 Apr 2012 01:04:15 GMT
Status: 200 OK
Content-Type: application/vnd.com.somesite.todos-v1+xml;charset=utf-8
ETag: "1f298572c3b90d4b4e9da5f25ea0ba9a"
Vary: Accept

<list xmlns="http://somesite.com/todo/schema/list"
  xmlns:atom="http://www.w3c.org/2005/atom">
  <todo>
    <description>Write up demonstration of to-do API</description>
    <atom:link rel="resource" href="http://somesite.com/todos/1023"/>
  </todo>
  <todo>
    <description>Add more to-do items</description>
    <atom:link rel="resource" href="http://somesite.com/todos/1312"/>
  <todo>
</list>

In this example the client requests the current list of to-do items from the server and specifies that it accepts responses that are specifically version 1 of our XML API. Since the server knows what version of the API the client is using, it’s easy to provide the response in the format that the client really wants.

The server provides an “ETag” header with the response, in this case this is a value that will change every time the list changes. We could hash the list of to-do items or the unique id of each hash and it’s version; whatever makes the most sense. The client can then issue a conditional request to see if the resource has changed instead of simply re-fetching all the data.

Lastly, the server returns a list of the current to-do items and a pointer to where the client can find our schema for this list. We’re also leveraging the Atom Publishing Protocol to provide annotated links since that’s easier than writing up our own specification. Each to-do item has a link that indicates where that individual item lives, if the client needs to update or delete an item it knows where to go.

Next we create a new item by posting to the “todos” resource. Our API didn’t provide a link for this since it’s the common REST idiom.

POST /todos HTTP/1.1
Host: somesite.com
Content-Type: application/vnd.com.somesite.todos-v1+xml

<todo xmlns:atom="http://somesite.com/todo/schema/todo">
  <description>Buy more coffee!</description>
</todo>

The server creates the new item and then provides an affirmative response. It also sends a “Location” header, this indicates the URL for the new to-do item. The client can use that URL to fetch, update or delete the item. Lastly, the response body contains the complete content for the new to-do entry.

HTTP/1.1 201 Created
Content-Type: application/vnd.com.somesite.todos-v1+xml
Date: Mon, 02 Apr 2012 01:06:23 GMT
ETag: "64d053d0bcf1b4de41f852509fd5ffc2"
Location: http://somesite.com/todos/1314

<todo xmlns:atom="http://somesite.com/todo/schema/todo">
  <description>Buy more coffee!</description>
</todo>

With the location of the new resource in hand, the client can update this new to-do item at will.

PUT /todos/1314 HTTP/1.1
Host: somesite.com
Content-Type: application/vnd.com.somesite.todos-v1+xml
If-Match: "64d053d0bcf1b4de41f852509fd5ffc2"

<todo xmlns:atom="http://somesite.com/todo/schema/todo">
  <description>Buy more coffee! And more creamer.</description>
</todo>

The client provides the entirety of the updated to-do item to that resource’s location (“PUT” always expects the entire resource). Note that the client provides the ETag of the resource in the “If-Match” header; if the to-do item has been updated by another client the ETag won’t match and the server will return a “412 Precondition Failed” instead of enacting the change.

The server then responds with a confirmation.

HTTP/1.1 200 OK
Content-Type: application/vnd.com.somesite.todos-v1+xml
Date: Mon, 02 Apr 2012 01:12:46 GMT
ETag: "9cc7dc7e5f4a81e3d8daf752a622e84d"
Location: http://somesite.com/todos/1314

<todo xmlns:atom="http://somesite.com/todo/schema/todo">
  <description>Buy more coffee! And more creamer.</description>
</todo>

And there you have it. The simple API above gracefully handles changes to the version of API if needed. Our URLs are provided with the relevant resource, hopefully cutting down on the number of URLs that end up hard-coded into the client. The client uses ETag data when updating resources to avoid conflicts. Our examples revolve around XML messages but there’s no reason this same service couldn’t also support JSON data.

Caching

In the example above, we didn’t do anything all that interesting to support caching of our API’s responses. Nonetheless, the list of to-do items and those items themselves could be cached somewhere between the client and the server; all “GET” requests are cache-able by default.

By adding headers to our responses we can exert some real control over how these responses are cached. We can specify a specific time, after which intermediate caches will need to fetch a fresh copy. We can specify what sort of cache is allowed to cache our data, indicating that the client can cache it locally but intermediate caches cannot. And, of course, we can tag certain responses as entirely uncache-able.

The client can save itself some transmission overhead by using conditional requests. In the example above, the client provided an “If-Match” header when updating a resource, ensuring that the resource was in the expected state. The client could use the very same logic when fetching the list of to-do items, it would only pull down the new list if it had changed.

Modeling a Protocol

We only covered some simple create, update and delete functionality in the example above. While this is a good fit for how most of us think about REST services it isn’t the whole story. This same method of including links with the resource can also be used to define a protocol, instructing the client on how a resource can be used. As the client invokes these URLs for a resource the state of the resource changes, the server instructs the client as to which changes are available by varying the supplied URLs.

There’s a great presentation out on slideshare by Dr. Jim Webber that walks through how such a protocol might work. His example is reasonably complicated, it works through ordering a coffee from coffee shop. He’s also written some interesting books on the subject.

You Don’t Need to Do it All On Your Own

There are some frameworks out there that make the construction of this kind of API much easier, Webmachine being the most popular example. These frameworks handle the lion’s share of the header parsing and exception handling for you, leaving you to implement only the functionality that makes your API work. In fact, I’m currently working on a library for Clojure that does just this (naturally, it’s a work in progress).

All Your HBase Are Belong to Clojure

2012-01-12T00:00:00-05:00

I’m sure you’ve heard a variation on this story before…

So I have this web crawler and it generates these super-detailed log files, which is great ‘cause then we know what it's doing but it’s also kind of bad ‘cause when someone wants to know why the crawler did this thing but not that thing I have, like, literally gajigabytes of log files and I’m using grep and awk and, well, it’s not working out. Plus what we really want is a nice web application the client can use.

I’ve never really had a good solution for this. One time I crammed this data into a big Lucene index and slapped a web interface on it. One time I turned the data into JSON and pushed it into CouchDB and slapped a web interface on that. Neither solution left me with a great feeling although both worked okay at the time.

This time I already had a Hadoop cluster up and running, I didn’t have any experience with HBase but it looked interesting. After hunting around the internet, thought this might be the solution I had been seeking. Indeed, loading the data into HBase was fairly straightforward and HBase has been very responsive. I mean, very responsive now that I’ve structured my data in such a way that HBase can be responsive.

And that’s the thing: if you are loading literally gajigabytes of data into HBase you need to be pretty sure that it’s going to be able to answer your questions in a reasonable amount of time. Simply cramming it in there probably won’t work (indeed, that approach probably won’t work great for anything). I loaded and re-loaded a test set of twenty thousand rows until I had something that worked.

Setting Up Your Environment
What is HBase Again?
Goals: Web Application for Log Viewing
Laying Out My Log Data
Loading Data into HBase with Hadoop
Package and Deploy
For Great Justice

Setting Up Your Environment

Both Hadoop and HBase have pretty decent documentation that goes over installing and configuring these tools. I won’t re-hash that here, especially given the variance in setup (single node for development verses multi-node for deployment). In my experience, getting a multi-node development setup with virtual machines (i.e. with Vagrant) was problematic; even with host names properly configured I had issues with connections timing outa.

I’m using Clojure 1.3, the libraries that I needed were available in Clojars but are built for 1.2. I forked these libraries and moved them to 1.3 to keep things neat for myself. If you’re using Clojure 1.3, you can clone these repositories and build your own copies.

Clojure Hadoop (or my fork for Clojure 1.3 compatibility)
Clojure HBase (or my fork for Clojure 1.3 compatibility)

My fork of the HBase library requires a 1.3 compatible version of the monolithic Clojure Contrib library. You can download a copy if you need it.

What is HBase Again?

HBase is provides a BigTable-like database on top of Hadoop. Uh-oh, I said database! Well, it’s not at all like a SQL database; it’s more like a really big sorted map. In the simplest scenario, given a particular key HBase can quickly give you all of the values associated with that key. Because it’s sorted, getting all of the rows between one key and another is pretty fast. You can also quickly get all of the rows that begin with a particular key.

While Hadoop will let you store huge amounts of data and run jobs to analyze that data, you need to do something with the results of those jobs. Often making those results available for people to look at is good enough, in my case the client wants to be able to search for specific information (for instance, the log records produced when a particular host was crawled) and this isn’t something I could do easily with Hadoop. My solution was to store the crawl logs on my Hadoop cluster and then run a Hadoop job that would load this data into HBase. HBase will let customer query for data on-demand.

This is important to note: it’s a good solution for myself and my peculiar scenario. There are other data stores that may be a better fit for someone else. I already have a Hadoop cluster up, running and in active use. If I was already using Riak, I most likely would have concentrated my efforts on making that work instead.

An HBase table is made up of rows, each row contains any number of columns and each one of those columns belongs to a column family. Each column family can then have a qualifier and a value. It took a bit of thinking to get my head around what a column family was, it’s purpose and how it should be used. Generally speaking, items in the same column family are stored together physically and that makes fetching all of the data for a particular row or group of rows column family faster. There’s a good discussion about how HBase actually stores this data in the O’Reilly HBase book. It’s definitely worth reading.

Goals: Web Application for Log Viewing

Before we move further down this path and try to figure out what data should go where, we’re going to take a brief detour and look at the web application that will be presenting the log data. In my opinion, this will help us get a handle on what data we need and, at least in terms of idealized application usage, when we will need it.

The data set I’m working with looks like this: a set of files, each line of each file contains one web crawler transaction. This will have the date and time of the transaction, the URL that the crawler tried to crawl, the response from the host serving that resource and the status the indexer assigned to the result. Here’s a sample row:

Timestamp	Response	Status	URL
2011-01-12-12:44:33	200	NEW	http://twitch.nervestaple.com
2011-01-12-12:45:12	200	NEW	http://twitch.nervestaple.com/index

The client wants the ability to see a summary page for each crawled host, this page should display the number of URLs for the host, how many returned content and how many had problems. They’d like to have a report page for each crawled URL that provides some summary information and a list of transactions for that URL. They want to be able to bang in an URL or host name into a search box and get dropped at the right page. They say that’s everything but I can’t help but have the feeling that this really only represents everything they’ve though of so far.

Laying Out My Log Data

Given the way that column families work, it makes sense to store data that we want to access all at once in the same column family. While it’s hard to know what data people will want, we can make some guesses. In my case, I know what my web application is going to look like and I can use that information to inform my choices.

First up, we want the ability to fetch all of the transactions for a given URL. This means that we’ll want an HBase table where the key will at least start with the URL. Each URL could be crawled multiple times, so we’ll combine the URL with the date and time crawled. It’ll look unpleasant but something like this:

http://twitch.nervestaple.com/index2011-01-12-12:45:12

HBase will store our data sorted by key, this means that when we want all of the transactions for a given URL it can jump to the first row that starts with our URL and then return each row there-after in sequence.

For our log data, we won’t have particularly interesting column families. I chose to name my columns families “request”, “transaction”, “response” and “crawler”. To follow through with our example, the data in HTable would look something like this:

Row	Column Family	Qualifier	Value
http://twitch…	request	url	http://twitch…
	request	host	twitch…
	transaction	date-time	2011-01-12-12:45:12
	response	code	200
	crawler	status	New

This layout makes it easy to pull out the rows for a particular URL. When we want to provide summary information for a specific host, this isn’t as helpful. While we could scan through all of the records for the host in question, the client would have to crunch through the returned data and calculate the summary data. This will be problematic in practice, instead we’re going to leverage another HBase function: counters.

HBase will let us treat a column as if it were a counter, this lets us atomically increment the counter in just one call to the server. As we load data into HBase, we’ll increment these counters; this will make it easy to provide the summary data the client demands. For instance, we’ll have a table called “host-stats” with the column families “transactions”. We can use a qualifier like “total” to represent all of the transactions for that host. Some example rows might look like this:

Row	Column Family	Qualifier	Value
twitch.nervestaple.com	transactions	total	104
bakery.somewhere.com	transactions	total	83

In practice it’d make sense to break this down further. We could have a family called “transactions-yearly” and use the year portion of the crawl date as our qualifier. We could have a family called “transactions-monthly” and use a combination of the month and year portions of the crawl date as the qualifier. You get the idea.

HBase will let us have as many items of data for each column family as we want, in our last example we’re using the qualifier to distinguish between each year and month combination in the “transactions-monthly” family.

Loading Data into HBase with Hadoop

There are a couple different ways to load data into HBase. Given the amount of computation we have to do (incrementing the counters based on host and URL) it makes sense to write a Hadoop map/reduce job to load in this data.

Setting Up Our Project

We’ll be using both the Clojure Hadoop and the Clojure HBase libraries. I’m going to assume that you’re using Leinigen to manage your Clojure projects. If not, you’ll want to revisit that. Go ahead a create a new project with…

lein new crawl-log-loader

Next up you’ll want to add the dependencies to your “project.clj” file. If you built the libraries from my repositories, you’ll want to add the following:

(defproject crawl-log-loader "1.0-SNAPSHOT"
    :description "Load Crawler Log data"
    :dependencies [[org.clojure/clojure "1.3.0"]
                   [clojure-hbase "0.90.5-SNAPSHOT"]
                   [clojure-hadoop "1.3.3-SNAPSHOT"]
                   [org.clojure/tools.logging "0.2.3"]
                   [org.clojure/tools.cli "0.2.1"]]
    :dev-dependencies [[org.codehaus.jackson/jackson-mapper-asl "1.9.2"]
                       [org.slf4j/jcl104-over-slf4j "1.4.3"]
                       [org.slf4j/slf4j-log4j12 "1.4.3"]]
    :main crawl-log-loader.core)

We’re defining our dependencies on Clojure, the HBase library and the Hadoop library. We need the Clojure “tools.logging” library so that we can log message while our Hadoop job is running. I like to include “tools.cli” so I can look up documentation for functions from the REPL while I work. Lastly, the Hadoop libraries depend on Jackson and SLF4J; they’ll be present on our Hadoop cluster but we’ll need them around in order to build our application.

Use, Require and Import

With the project setup, it’s time to add some code. Open up the core file (src/crawl_log_loader/core.clj), and add something like this…

(ns crawl-log-loader.core
  (:use [clojure.repl]
        [clojure.tools.logging]
        [clojure.tools.cli])
  (:require [clojure.string :as string]
            [clojure-hadoop.gen :as gen]
            [clojure-hadoop.imports :as imp]
            [clojure-hbase.core :as hb])
  (:import [org.apache.hadoop.util Tool]
           [org.apache.hadoop.hbase.client Increment]
           [org.apache.commons.logging LogFactory]
           [org.apache.commons.logging Log]
           [java.net URL]
           [org.apache.hadoop.hbase.util Bytes]))

I’ve added the Clojure REPL, logging and CLI tools to make it easier to bootstrap your application and parse out command line arguments. Since we’re parsing log files, we’ll need the Clojure String library as well. After we pull in our libraries for Hadoop and HBase, we import some of the Java classes we’ll need to make those libraries work.

I haven’t mentioned if before, but HBase stores everything as an array of bytes. It has no notion of type, it just sees byte arrays. Our last import provides a utility class that makes it easier for us to convert objects like strings and numbers into byte arrays.

Define Our Hadoop Job

We’re going to do this a little backwards and add the function that sets up the Hadoop job next. This function will reference our map and reduce functions although we haven’t written those yet.

(defn tool-run
  "Provides the main function needed to bootstrap the Hadoop application."
  [^Tool this args-in]

  ;; define our command line flags and parse out the provided
  ;; arguments
  (let [[options args banner]
        (cli args-in
             ["-h" "--help"
              "Show usage information" :default false :flag true]
             ["-p" "--path" "HDFS path of data to consume"]
             ["-o" "--output" "HDFS path for the output report"])]

    ;; display the help message
    (if (:help options)
      (do (println banner) 0)

      ;; setup and run our job
      (do
        (doto (Job.)
          (.setJarByClass (.getClass this))
          (.setJobName "crawl-log-load")
          (.setOutputKeyClass Text)
          (.setOutputValueClass LongWritable)
          (.setMapperClass (Class/forName "crawl-log-loader.core_mapper"))
          (.setReducerClass (Class/forName "crawl-log-loader.core_reducer"))
          (.setInputFormatClass TextInputFormat)
          (.setOutputFormatClass TextOutputFormat)
          (FileInputFormat/setInputPaths (:path options))
          (FileOutputFormat/setOutputPath (Path. (:output options)))
          (.waitForCompletion true))
          0))))

This article isn’t about parsing command line arguments, but the above is a good habit to get into. We use the CLI library to both setup our arguments and to parse those arguments out into a hash-map. More information on how this library works can be found on the project’s page.

Hadoop wants our application to return a status code that indicates healthy completion of the job or exiting under an error condition. We return “0” to indicate that our job exited normally. In real life you may want to do something more clever.

If our app isn’t invoked with the “-h” or “–help” flag, we setup the Hadoop job. We create a new Job object and set a bunch of fields. Note that we set the output key and value class. The main purpose of this job is to load data into HBase but we’ll also output the number of transactions per host. This could be used for any number of things, perhaps we want to double-check the data stored in HBase.

We set the mapper and reducer classes, we’ll write those up next. We set the input and output formats; the TextInputFormat reads plain text files line-by-line, a good fit our log input. The TextOutputFormat writes plain text files.

Mapping Function

We’ll now add our mapping function. Make sure that you add your own mapping function above the definition of your “tool-run” function.

(defn mapper-map
  "Provides the function for handling our map task. We parse the data,
  apply it to HBase and then write out the host and 1. This output is
  used to provide a summary report that details the number of URLs
  logged per host."
  [this key value ^MapContext context]

  ;; parse the data
  (let [parsed-data (parse-data value)]

          ;; apply the data to HBase
          (process-log-data parsed-data)

          ;; write our counter for our reduce task
          (.write context
                  (Text. (:host parsed-data))
                  (LongWritable. 1))))

This isn’t so tricky! We read in a key and a value, they key will be the line number of the file being processed and the value will be the text of that line (a log entry). We don’t really care which line of what file this entry came from so we ignore it. Then we parse the line of log data into a hash-map, apply that data to HBase with our “process-log-data” function (yet unwritten) and then write out data for our final report.

We’re writing out the host for the URL that was crawled in this log entry and the number 1. During the reduce phase we’ll sum the values for each host and output the total number of transactions. In fact, let’s do that right now.

Reduce Function

(defn reducer-reduce
  "Provides the function for our reduce task. We sum the values for
  each host yeilding the number of URLs logged per host."
  [this key values ^ReduceContext context]

  ;; sum the values for each host
  (let [sum (reduce + (map (fn [^LongWritable value]
                             (.get value))
                           values))]

    ;; write out the total
    (.write context key (LongWritable. sum))))

Again, this function isn’t scary at all. We map over the incoming values (Hadoop wraps the number in a LongWritable instance), pull out the actual values and reduce those values into our final sum. We write out the key, which is the name of the host and the sum, the total number of transactions for this host.

We’re nearly through, the last function is the bit that applies our data to HBase.

Load Data into HBase

We need to parse out our line of log data into something easier to work with, a hash-map. My files are separated with spaces making this very easy.

(defn parse-data
  "Parses a String representing a row of data from an ESP crawler log
  into a hash-map of data."
  [text]

  ;; parse out the row of data
  (let [data (string/split (str text) #"\s+")]

    ;; return a map of data
    {:timestamp (nth data 0)
     :reponse (nth data 1)
     :crawler-response (nth data 2)
     :url (nth data 3)
     :host (.getHost (URL. (nth data 3)))}))

This is a bit simplistic, but you get the idea. In practice you’d probably want to be more careful and make sure the input data is valid. You’d likely want to split the date and time out or even parse it back into a real date instance.

Lastly, we want to add this row to HBase and update some counters.

(defn process-log-data
  "Handles the processing of log data by applying the map of data to
  the proper counters and adding the correct rows to our HBase
  tables."
  [parsed-data]

  ;; add our row of data
  (hb/with-table [urls (hb/table "urls")]
    (hb/put urls
            (str (:url parsed-data)
                 (:timestamp parsed-data))
            :values [:request [:url (:url parsed-data)]]
            :values [:request [:host (:host parsed-data)]]
            :values [:transaction [:timestamp (:timestamp parsed-data)]]
            :values [:response [:http (:response parsed-data)]]
            :values [:response [:crawler (:crawler-response parsed-data)]]))

  ;; update our host stats table
  (hb/with-table [host-stats (hb/table "host-stats")]
    (.incrementColumnValue host-stats
                           (hb/to-bytes (:host parsed-data))
                           (hb/to-bytes "transactions")
                           (hb/to-bytes "total")
                           (.longValue 1))))

We’re keeping this simple so that you get an idea of how this works. We add the row of data to our “urls” HBase table, then we increment to total number of transactions for this host in the “host-stats” table. We don’t have to worry if there’s already a row in the “host-stats” table for this host, if there isn’t then HBase will create a new row with a value of zero and then increment it by our supplied value, 1.

Package and Deploy

Deployment is straightforward, create an “uberjar” with Leiningen and then copy that out to your Hadoop cluster. From there you can invoke the JAR with an input and output path.

lein uberjar
...watch Leiningen work...

scp crawl-log-loader-1.0-SNAPSHOT-standalone.jar hadoop1.local:/hadoop

ssh hadoop1.local
...log into your Hadoop node

And then run your job. It turns out this is trickier than you’d think. If you simply invoke the JAR with the “hadoop” command it will run on the local job runner (it won’t run distributed across the cluster) because it won’t be able to find your HBase install. To fix this, create a new folder in your project directory called “resources”. Files in this folder will be bundled up by Leiningen into your final JAR and they’ll be present on the class-path when the application is launched.

Next, create a file called “hbase-site.xml” in this folder. It should look something like this:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>hadoop1.local</value>
  </property>
</configuration>

The Zookeeper Quorum value should list all of your Zookeeper nodes, if you’re running in standalone development mode then there’s just the one. If you are running in production this should be a comma separated list of host names. Lastly, you can use a script similar to the following to start your job.

export HBASE_HOME=/hbase
export HADOOP_HOME=/hadoop

HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` \
  ${HADOOP_HOME}/bin/hadoop jar crawl-log-loader-1.0-SNAPSHOT....jar \
  -p hdfs://hadoop1.local:54310/data/in \
  -o hdfs://hadoop1.local:54310/data/out

This sets up environment variables that point to our HBase and Hadoop installations. We then load our application in the context of our Hadoop instance and add our HBase installation to the class path. The “-p” and “-o” flags at the end are interpreted by our application.

For Great Justice

While this isn’t production code it should be enough to get you started. We now have some data loaded and that will make it a lot easier to explore HBase and evaluate the product. Once again, I am amazed at how easy it is to work with these frameworks using Clojure! A big thanks goes out to the developers and maintainers of the Hadoop and HBase libraries that make this all so easy, as well as the developers and maintainers of Hadoop, HBase and Clojure themselves.

Java Doesn't Need to Be So Bad

2011-12-04T00:00:00-05:00

I do a lot of Java coding and I enjoy it. I admit that there is a lot of typing, often a lot of boilerplate and getting even simple tasks done can involve too much work. Most of the tools that try to fix these problems trade one moment saved for another lost. Maven's XML based configuration file is a good example: Thank you for making my project easier to manage and I won't forget that you made me edit XML to do so.

These are the things that you live with, these are the things you trade for using a language that thinks it's finishing what C++ started; a language that is loved equally by mammoth corporate IT departments, mammoth corporate IT vendors and mammoth corporate IT consultants. What you don't need to tolerate is this tendency to crazily over-engineer even the simplest solutions.

We've all Seen It: The Factory Factory
I Point the Finger at You, I Have Already Pointed it At Myself
Premature Optimization and Re-factoring
Moderation in All Things

We've all Seen It: The Factory Factory

It's a long-standing joke because we've all seen it and we've all grown to hate it: "The Abstract Factory Pattern," euphemistically referred to as "The Factory Factory". As envisioned by the Gang of Four, this is a handy way to create families of related objects without needing to know much about their concrete implementations. It's a clever idea and when we see it used in a library or if we end up coding something like this ourselves, our reaction should be "Yes, very clever!" So why are we so often complaining?

I think we're having reactions like this because when we look at an API that does this, it's often far too complex; it's overkill. If you're using a library and you are noticing this pattern, as often as not it turns out that the author of said library is doing it wrong: there's a factory that returns a factory that returns one implementation. One factory is a bad sign, that the one factory has only one implementation to build tells us that this pattern isn't applicable. If you are working with a library that uses this pattern and it's not saving you a headache by hiding some obnoxious suite of not quite interchangeable objects, then this pattern is wasting your time and effort for absolutely no reward.

In case that you don't do a lot of Java coding and have no idea what I'm talking about, take a look at this bit of Java code that does just this: one abstract factory extended by two implementation factories. I'd have written example code myself but even that would be too much work. Note how silly and pointless this example is: it needs to be that way because if it were more true-to-life, you'd have a really hard time following it. Here's a more real-life example, notice how boring and co-dependent these classes are; this is the kind of problem this pattern is trying to solve.

I blame Sun, I blame your boss's boss, I blame unimaginative managers everywhere who looked at Sun's example code and thought "Yes! Our code needs to look like this too!" Because it doesn't and 99% of the time it really shouldn't. If you're starting a new project and you decide that the first step is to draw a really big UML diagram and then try to isolate key places to implement this or that design pattern then you, my friend, are also part of the problem.

I Point the Finger at You, I Have Already Pointed it At Myself

It could be that everyone my age, or everyone who had a similar career path, or everyone that read documentation from Sun when they were too young to know how horrible such documentation actually is have all ended up making these same mistakes. I toiled away on a project for four long years that heavily leveraged the now universally despised EJB 2 specification. I myself have drawn out huge UML charts that ended up being printed as a shockingly large sheaf of pages that were then taped to a whiteboard. I personally have to live with the shame of a very clear and vivid memory wherein I circled objects that covered nearly four sheets of this paper and wrote, in my own hand and with a clear-head: "A good candidate for an abstract factory! :)"

Eventually I realized that the emoticon was mocking me. Laughing at my presumptuousness, laughing at my premature optimization. Laughing at my fear of re-factoring.

Premature Optimization and Re-factoring

That's what I was doing, I was trying to solve a problem before I even had it. Before I had even written one line of code, I was trying to decide how the pieces of the puzzle fit together. It was simply too much detail too early on. No one ever gets it right the first time, what made me think that I was so special?

I'm not saying it's not worth it to think ahead and to plan, but be mindful that no one has perfect knowledge of their problem space and just as things that seem simple can turn out to be complicated, sometimes problems that seem really hard end up solving themselves. Maybe that's just how business logic works: you talk to the right person and it turns out that you had it wrong the whole time.

Moderation in All Things

I'm advocating an approach where you're mindful of these design patterns but you keep them in the background. When you have working code that is starting to get messy, or when someone on your team has been banging their head on a problem for way too long, that's when you take out the patterns book. Because at that point you have a pretty good idea of what the problem actually is. You even have a solution that's mostly working and that solution can be re-factored to fit the pattern that seems most appropriate.

You probably won't ever end up with a factory that creates a factory that creates this one object that people only ever use once because that is plainly wrong. But if you ever do code up an abstract factory, when people end up interacting with your API and they whine about the factory factory you can confidently tell them that the alternative is much, much worse and show them how you've actually made their life much easier.

Blogging With Emacs

2011-10-09T00:00:00-04:00

I don't want to waste time explaining why I use Emacs all day, every day, and eschew more traditional development environments. Either you already use Emacs and I'm preaching to the choir or you don't and are very unlikely to read this article. I will keep it short: Emacs is the easiest for me to customize, changing the way something works often requires nothing more than adding a couple lines of code and then the thing does what I want. It's like an amorphous blob that needs to be shaped into the tool that you need but once so formed, that tool is thereafter only one click away.

Org Mode
The Workflow
Setting Up a Publishing Project
- Setup your Weblog Directory and Jekyll Template
- Setup your Org Mode Publishing Project
Writing Your First Article
Publish Your Weblog
Greatness Is Now Expected

Org Mode

I use Org Mode when authoring all of my posts. I let Org Mode handle exporting my completed posts to HTML, if you want to follow along at home you'll need to have Org-Mode installed and be fairly comfortable using it.

If you use Emacs for any sort of meaningful work and you're not actively using Org Mode, you need to revisit this as soon as humanly possible. It's conceivable you have your own system, carefully crafted and then fine-tuned over the years into a razor sharp knife that can disembowel the dreary red-taped monster of your work-life, spilling out the most crucial and important information onto the floor so that you can nudge and prod it with your booted foot, but I sincerely doubt it. It's more likely that you have information spread all over your desk, your laptop, your office, perhaps your office floor. Post-it notes, the back of errant print-outs so lengthy that you feel guilty eventually (and inevitably) dropping them in the recycling bin, well-meaning "Get Things Done" type applications that strand your critical data on attractive yet inaccessible prison islands. Hear me now: stop whatever your doing and setup Org Mode, then take the time over the next couple of days to migrate all of your to-do items and notes. Trust me, it's the right thing to do.

The Workflow

Org Mode provides the ability to export your files one at a time, but you can also export them as an entire project. The entirety of my weblog is contained in one project, once I've completed a new article and edited it to my satisfaction I simply export the "weblog" project to the directory that contains my completed blog.

That brings us to the second component of this solution, Jekyll. With Jekyll, we can take a directory of articles and transmute them into a complete and cohesive website. Jekyll will scan through the posts and construct the website, ready to publish, in a sub-directory called "_site".

Lastly, with the completed website in hand, you need to get it out there on the internet. The simplest solution would be an Rsync script to copy new files out to your web space.

Setting Up a Publishing Project

Before we go any further, I need to let you know that this is not a trivial task. It's not as bad as, say, setting up a new web-server from scratch but it's definitely more involved then simply creating a new account at Wordpress.com. I've been operating under the presumption that you take your blogging much too seriously to be satisfied with banging the text into the kind of brain-dead web-form served up by most weblog products. If this is not the case, please forgive the misunderstanding and be on your way.

Setup your Weblog Directory and Jekyll Template

The first step is to setup a directory where you will store your weblog. I store mine in the root of my home directory in the aptly named "weblog" directory. You are free to use the same name and location.

Jekyll reads through the contents of a folder called "jekyll" for information about your blog, templates, and various bits of information. It's possible to start with a nearly empty "jekyll" directory but that is absolutely no fun. Instead we'll use a template that Christian Hellsten has put together. Download the ZIP archive of the project from the following link:

https://github.com/christianhellsten/jekyll-template/zipball/master

Go ahead and unzip the archive, it'll be named something like "christian-hellsten-…" Rename the directory to "jekyll" and place it in your weblog's directory.

The next step is to create a folder where you'll store the Org Mode files that contain your posts and the contents of the various pages. Create another directory in your weblog folder and call it "org".

Lastly, you'll want to customize the "_config.yml" file in your brand new "jekyll" directory. Leave the "url" alone for now but go ahead and change the title, description, etc.

Setup your Org Mode Publishing Project

In addition to doing very nearly everything it can to help you organize your notes and tasks and generally keeping you on top of the massive, sprawling mess that is your life, Org Mode can also export files as HTML. You can do this file-by-file as the need arises or, and this is what we'll be doing, you can setup a project and export the entire thing all at once.

Open up your appropriate Emacs configuration file (you do have more than one, don't you?) and add the following stanza to the end.

(setq org-publish-project-alist
      '(("org-weblog"
         ;; Path to your org files.
         :base-directory "~/weblog/org/"
         :base-extension "org"

         ;; Path to your Jekyll project.
         :publishing-directory "~/weblog/jekyll/"
         :recursive t
         :publishing-function org-publish-org-to-html
         :headline-levels 4
         :html-extension "html"
         :body-only t)

        ("org-static-weblog"
          :base-directory "~/weblog/org/"
          :base-extension "png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf"
          :publishing-directory "~/weblog/jekyll/attachments"
          :recursive t
          :publishing-function org-publish-attachment)

        ("weblog" :components ("org-weblog"
                               "org-static-weblog"))))

This describes two tasks, one will use Org Mode's HTML extension to publish all of your ".org" files to the "jekyll" directory. It will only write the body of these files, they won't be complete HTML pages. This is what we want, Jekyll will template these pages and wrap them up into final, web-ready content.

The second task moves any other files that we might have out to the "jekyll" folder. This includes things like pictures or charts or whatever content you've linked into your Org Mode files. Ideally these links will work when they come out the other side of the content grinder that is your website but your mileage may vary.

With this in place, evaluate that function so that Emacs is ready to publish your brand new site.

Note: An Odd Issue

Recently, for now reason I can discern, publishing my weblog project would fail after attempting to export the first page to HTML. Org Mode would export the file to HTML and then copy the exported text to the clipboard, this should be text but it wasn't and this was breaking the project export process.

Adding the following line to my Org Mode setup fixed this issue. What would I do with that buffer of HTML anyway?

(setq org-export-copy-to-kill-ring nil)

Writing Your First Article

The stage is now set for your new adventure in authoring. In real life it will likely take days and days of hand-wringing and second guessing just to get a rough draft out and that will inevitably be followed by another several days of tireless editing and pointless yak-shaving while you get it "just right". We'll be putting such gritty reality aside for now and, instead, writing up a test article completely devoid of any useful information.

Naturally we need a place to store your articles. Create a new directory called "_posts" in the "org" folder that lives in your "weblog" directory. Pull up Emacs and create a new file called something like "2011-10-09-test_blog_article.org". At this point you'll have a new, pristine Org Mode buffer spread before you.

"none"body

Note that the title in the Org Mode header is just for you, Org Mode and Emacs. Jekyll won't see it, it will ignore the entire Org Mode header.

The text in the first HTML stanza is used to tell Jekyll some of the details about this post, this is called "front matter". More information on how this works is available on the Jekyll wiki. For now, just paste this text in so we can move on.

Publish Your Weblog

With your first article written, we're ready to publish your blog! Invoke the "org-publish" function and Emacs will prompt you for the name of the project to actually publish, type in your project name (likely "weblog") and press return. Lean back as Org Mode dutifully exports all of your content to HTML.

Now that the content has been converted from the versatile Org Mode format into the unpleasantness that is HTML, you can turn your weblog on locally to see what it looks like. Pull up the command prompt and navigate to your weblog's "jekyll" directory and start-up your own little web-server like so…

rake server

Jekyll will be invoked and it will create an entire static site in the "_site" folder. A web-server will be spun up to server those files and you will be able to view your new blog at http://localhost:4000. Well done!

Greatness Is Now Expected

With the bare bones of your website now constructed, it is up to you to generate the quality content that only you can generate. To get your weblog up on the World Wide Web, you'll want to copy the contents of the "_site" folder to the appropriate directory on your web-server. You could setup something fancy like getting it all in Git and using a commit hook or use rsync to copy the changed pages out. It's up to you!