Building MutualFund data via Zerodha’s API

7 min readMay 23, 2022

Problem Statement

Unable to build a catalogue of mutual funds for my project as no vendor provides a list consisting of details that can be used as a catalogue for free.

What’s the project?

A few Moons back I started constructing a P2P lending platform where folks join as a group and pool their money and a representative uses the collected fund to invest in Stocks, Crypto, Precious Metals and other ways for money to grow. Find the ERD of the project and code here.

We have a catalogue for cryptocurrencies as many third parties provide for them but nothing such as mutual funds. So we made an inhouse solution using Zerodha’s Kite APIs

How Zerodha is linked

Zerodha did a great job by providing a few APIs using the Kite platform, these APIs are great for fetching fund details, portfolio analysis, mutual funds data, etc.

Why not use Zerodha’s API for mutual funds data?

Well, the problem is very simple and plain, the data is sent in CSV format. This means there are some around 5k data present in CSV, one way to navigate is to render this on the frontend anyhow, well this is possible but how to apply filters, sort or in fact pagination in this case?

Sample data from the Zerodha websitetradingsymbol,amc,name,purchase_allowed,redemption_allowed,minimum_purchase_amount,purchase_amount_multiplier,minimum_additional_purchase_amount,minimum_redemption_quantity,redemption_quantity_multiplier,dividend_type,scheme_type,plan,settlement_type,last_price,last_price_dateINF200K01LZ0,SBIMutualFund_MF,SBI Liquid Fund,0,1,5000.0,1.0,5000.0,0.001,0.001,idcw-reinvest,Debt,regular,T1,1091.425,2022-05-20INF200K01MA1,SBIMutualFund_MF,SBI Liquid Fund,0,1,500.0,1.0,500.0,0.001,0.001,growth,Debt,regular,T1,3325.6891,2022-05-20

So we had to navigate our way around it. Why not do some in house construction of this chunk of data?

Possible ways of inhouse solution

1. One of the ways is to store the CSV on the server and search through the file, do pagination in the same way, and filter the data accordingly.

A sample of how data can be filtered for a particular item (assuming third element of the csv file is for mutual fund scheme type) for(let row of textContent.split(“\n”)){
    const rowItems = row.split(“,”);
    if(rowItems[2]=="equity"){
        excuteSomeCode
    }
 }

Pros:

Data can be filtered, sorted and can be paginated for a huge chunk of data.
No extra time is needed to update the data, daily call the download API once and get the CSV data for the day.

Cons:

Too much complexity to handle for calculation of data, including filtering and sorting them, a lot of head smashing on the keyboard or reading the documentation for hours.
The burden on server memory by the file and old data is lost forever after refreshing. Hence no graph of the NAV bar can be made.
CSV will be built on every call to the server, making it a source of memory leaks(one fix is to store it once and use it for all calls).

2. Storing data in a database and fetching it for subsequent calls

Pros:

Data can be traced and old data can be used for creating graphs.
No-load on the server and follow a clean coding practice.

Cons:

Updating the data takes a lot of time, each 5k entry is processed and modified in the database, if the server breaks in the middle while updating, old data will be displayed.

What did I follow?

Well I went for the second flow, here’s what I did:-

Download the CSV file from Zerodha’s website.
Traverse through the CSV.
Upsert the data in MongoDB.
Fetch the catalogue for filters, sort and pagination.

So challenge over? We’ve perfect data?

Well no, this was the tip of the iceberg, the coding part was yet to come. Before I proceed with the final code I will have to clarify the choice of tech I made and why?

I used MongoDB as my database for storing the CSV data from Zerodha, the reason being access to the cloud(for free data is secure somewhere and available), dynamic schema if I add a few new params in future(to be discussed what I intend to add).

Using Nodejs to make the backend. But why Nodejs? personally, I’m a big fan of Golang and have been coding for a long in GO yet I chose to continue in Nodejs for this project. Similar database interaction performance, better community support and of course the almighty Nodemon.

Code part I feel made it easier from me to process

https.get("https://api.kite.trade/mf/instruments",async (res) => {
  const path = `./app/Controllers/img.csv`;
  const filePath = fs.createWriteStream(path);
   res.pipe(filePath);
   filePath.on('finish',async () => {
   readFile(`./app/Controllers/img.csv`, "utf8", async (error,       textContent) => {
   if(error){ throw error; }
    for(let row of textContent.split("\n")){
       const rowItems = row.split(",");
       await this.repository.bulkUpsertMutualFundData(rowItems)
    }
    }
})
  filePath.close();
  await unlinkAsync(path);
})

Trying to break down the system I’m trying to build:-

HTTPS.get() will get the data from the website, i.e, download the CSV on the server.

Why not convert this callback to a promise and get the response after the update is done? Because this will hold our response for almost 30 mins, so better to leave it to update the data and move ahead.

2. We store the CSV at a particular location in the server and continue to process it.

Instead of using fs.writeFile I went for fs.createWriteStream(path) as it's a smarter way for a huge amount of data.

What does createWriteStream do? well, it returns a writable stream object for the file at path. But what is a Stream? Well, truth be told I wasn’t aware of this wonderful thing a few days ago(I wasn’t processing files in my previous projects and truth be told this is a wonderful tool to be used for processing files).

So what is Stream? Streams are objects that let you read data from a source or write data to a destination in a continuous fashion. To send it to a destination we can use Piping(Piping is a mechanism where we provide the output of one stream as the input to another stream). For a better tutorial click here.

const filePath = fs.createWriteStream(path); reads data from the source
res.pipe(filePath); pipes the stream to res

3. Using the EventEmitter ‘finish’ to read the CSV file row by row.

readFile(`./app/Controllers/img.csv`, "utf8", async (error,        textContent) => {
   (reads the file at the location)
    if(error){ throw error; }
    for(let row of textContent.split("\n")){
       const rowItems = row.split(",");
       await this.repository.bulkUpsertMutualFundData(rowItems)
    }
  }})

Great! so our data is sent to the database, row by row upserting the data in the database? So let’s end it? I guess not, we haven’t closed the stream and the CSV file is still available on the server consuming memory.

filePath.close(); (closes the stream)
await unlinkAsync(path); (promisify the unlink function and move only when the file is deleted)

So all the pieces landed on their location on the board, and we have a clear way of storing data and make own Mutual Fund catalogue using Zerodha’s API.

This update of mutual fund API has to be called once per day as the NAV bar changes at the end of the day.

What we expect Portfolio to go vs how it goes

What’s more to do? what will I do next?

If you ever visited any website that shows mutual fund data you would see they show a graph of NAV bars, so the next target would be to fetch historical data of the navbar and show other details of the mutual funds in the project.

Also, try to add a stock catalogue option for stocks, sort mutual funds on their returns, etc.

Summary of things I did

Use Streams in nodejs
Process CSV data and store it on the cloud for any processing and build catalogue page.
As a final result, I have an API that would give me a Mutual Fund catalogue depending on my parameters and all I’m doing is processing a CRUD API for the details.

Things to do after reaching here

Read about Streams, and use Zerodha’s super cool APIs for building projects.

Building MutualFund data via Zerodha’s API

Pros:

Written by SHIVAM SOURAV JHA