Searchable Academic Database for Asian Ethnology

Problem

I have worked with Asian Ethnology as the Book and Film Reviews editor since 2017. The journal editors have constantly expressed their desire for a searachable database that allows users to search the journal content. I decided to develop this database to help them achieve this.

Approach

My plan was to deploy this as a MERN stack application. I like using the MERN stack because it flexible and quick to get set up. Since this isn't a project that is too complex, I thought this would be a great approach.

After setting up the full stack architecture in Node and React, I used BeautifulSoup to scrape journal data into a CSV file, which I then populated the database using csv-parser and basic mongoose. I also indexed the contents using a Schema, and implemented security measures to prevent sanitze data, enforce rate-limiting, and prevent XSS attacks.

The initial set up went smooth, so I deployed this to Render. The deployment went well, but the intial search for articles in my MongoDB database was always very slow. I try to adjust configurations and searched for the problem, but unless I upgraded to Render's paid plan, I don't think I would be able to address the slow initial load time.

Slow loads lead to very bad user experiences, and the almost 30 seconds it took to load the initial search results was not acceptable. As a workaround, I decided to refactor the app as frontend only and store the data in a JSON file instead of a MongoDB collection. Since the data being accessed and stored was not sensitive, I felt this was a reasonable alternative. I planned to deploy this on Netlify, which I know is fairly secure. I still took precautions and wrote them into the app for added security. This process took a few hours.

The searchable database is now live on Netlify. We plan to scale this a bit, adjust the UI, and add some more features to better observe trends over time in the future.