For the past couple of weeks I have been working on my first new feature for the Firefox Marketplace. It has been a nice change of pace to the usual bug-tracking-down-and-fixing cycle that I have come to know and love. It has also presented a slew of new challenges that have been both satisfying and frustrating all at the same time.
Ask and You Shall ReceiveI started out as I usually do by scanning the list of the most recent bugs added to Bugzilla. I came across one that was created by one of the core Mozilla developers that immediately peeked my interest since it was not a typical bug fix and it seemed relatively manageable with my current knowledge of the project. Currently, the Marketplace will recommend new applications for people to try based off of applications that they have already installed in either their browser or on Firefox OS. The idea behind this new feature is that if we also track when users view the details page of applications, we can use that information in the recommendation engine in order to suggest other applications that they may be interested in. In this way, even if someone has yet to install any applications, it may still be possible to make recommendations based on anything they have viewed.
I messaged the developer on IRC who made the feature request and asked if it would be okay if I tackled it and he was very open and even suggested that we reconvene during work hours to hash out a plan. Once I contacted him again, he laid out a few distinct steps of what needed to be accomplished. I had him assign me to the bug on Bugzilla and I got right down to work. It was a great change to have a developer be so receptive to helping out. He was by far the friendliest person that I have come into contact with so far during my open source work.
After ensuring my fork was up to date, my first step was figuring out how to create a new MySQL table. As I mentioned in one of my previous posts, Mozilla uses a lightweight schema management system called Schematic to handle database migrations. I added a new migration, which is essentially a
.sql file that consisted of a
CREATE TABLE statement along with some additional statements to create appropriate indices. The table to be created was to hold a relationship between users in the Marketplace and a particular app. The idea was that whenever an authenticated user visited an app’s detail page, a row would be created in the table.
After trying to run my migration, I found that I was getting an error since I didn’t rename one of the indices that I was adding. Upon changing the index name, re-running the migration failed again since the table had been created in the previous run. My final solution was to delete the preliminary table and run the migration once more using:
I created a new Django model which would be used to perform queries on the table that was just added and I developed some simple logic to create a new model instance whenever the app details page was loaded. The only trick at this point was to ensure that a user was actually logged in since it wouldn’t do us any good to store views that were performed anonymously.
I searched for the appropriate place to add some new unit tests and I found it within the fireplace app in Zamboni. I ran the existing test suite first to ensure that all of the tests were passing properly. I wanted to be certain that all of my tests were passing after the last fiasco. Initially I tried to add my new tests to an existing test suite but I was having some difficulties getting them to play nicely so I opted to just create a new test suite that was dedicated solely to my added tests. After all of my tests were green, I opened my pull request and anxiously awaited feedback.
After a short wait, I had received some comments from three different Mozilla developers. A couple of them didn’t seem to agree with the approach of storing the user and app relationship data within the MySQL database (even though this is what the original developer suggested). They thought that it might make more sense to store the data within some sort of key-value store. While this does make sense, we will want to make sure that the data is persistent so an in-memory key-value store would not suffice.
From more of the feedback, I learned that the current method that was being used would simply not work properly in production. The main issue is that in the production environment, a lot of caching is going on so that we are not required to hit the database whenever someone visits the details page of an application. This means that the block of code that I was using in order to track the views would not be called in all situations (in fact, it would be called in very few situations). Although I felt a little defeated at this point, an alternative was suggested which would consist of creating a new API endpoint that would be solely responsible for tracking the detail page views. We would be able to call this endpoint asynchronously from the Marketplace front-end, Fireplace, after everything has already loaded to ensure we aren’t slowing down any existing calls. This seemed to make a lot of sense and after chatting with the original developer we decided that this would be the best way to move forward.
If At First You Don't SucceedOnce my original pull request had been closed, I started to scope out different pull requests where API endpoints had been added in the past. I find that searching out existing code in a project is often the best way to learn quickly. I found one pull request in particular that proved to be a big help in understanding how exactly all of the API views and URLs worked together. I noticed that Django-Rest-Framework (DRF) was being used to provide a lot of the foundation for the API so I read quite a bit of the documentation there in order become a little more familiar with what was happening.
From the examples that I had found, I was able to add a new
APIView which would lookup the app based on the primary key in the URL and also retrieve the user. I then add a row in the new table for that app/user combination if it does not already exist. I opted to make this a
POST request since new data would potentially be created. I also assigned this
APIView to a new URL within the Fireplace application. Being strictly API-based, I was able to manually test my addition using a simple
curl call as illustrated below:
curl -X POST http://0.0.0.0:8000/api/v1/fireplace/app/5/details-view/
I revamped my test suite slightly by adding a new test that would ensure that everything would still work properly assuming that the API was called with no user logged in. I opened a new pull request with all of these changes. I quickly received a piece of feedback from one of the developers who again questioned the fact that we are using a MySQL table for storing the detail view data. He suggested using a simple log file to store the information but I will wait and see what the developer I am working with recommends on that front.
Even though my pull request was open, my job was not quite finished yet for this particular feature. Since the new implementation was entirely API-based, this meant that at some point we actually need to hit the API to perform the tracking. The Firefox Marketplace project actually consists of both a front-end and back-end component. The front-end, dubbed Fireplace, is responsible for making calls to the back-end for the required data and then properly displaying it in an appealing manner.
While I am hoping that these new pull requests will be merged this time around, I won’t be too upset if they find more show-stopping logic flaws. On both occasions, I have learned a lot of different things and as I approach the halfway point of the semester, I am thrilled with the progress that I have made. On the plus side, even if I have to redo these pull requests; third time’s a charm… right?