ERPNext CRM implementation at Thank U Foods — a review
I originally wrote this article last October, after a project I worked on at Thank U Foods, a subsidiary of the NGO Indian Association for the Blind. When it was completed, I felt I didn’t emphasize the good bits of Frappe enough. I intended to rewrite it, but with one thing and another, a year’s over and I haven’t modified this.
I’ve published it now so that people can see some of the limitations and unique features of Frappe, but I must warn that this is outdated — for example, one of the bugs (about scrolling down a list in the docs) I’ve listed below has since been fixed. Furthermore, a new version of ERPNext has been released, improving many aspects which I’ve covered in this article.
So use this to get an idea of what ERPNext and Frappe is before (or while) jumping in and using it, but take everything you read with a pinch of salt.
Description
This project’s goal was to get Salesforce functionality into ERPNext’s CRM, and after that, adding our custom requirements. We closed this project after five weeks because we felt that it was too big a project, and that, consequently, we couldn’t complete the goal in the required time.
The challenges we faced
To start, ERPNext is modeled very differently from Salesforce. In Salesforce, you had the Lead and Contact models. A lead is a potential customer, and a contact is a person who has made at least one purchase. And then you had the Campaign model. A campaign is used to attract new customers from leads and for customer retention (to make customers come back to the business). For example, we could email all the customers who did not buy in the last month. That would be an email campaign.
A lead or customer (for convenience, collectively termed as a member) could be in multiple campaigns. Salesforce had the option to have email, telemarketing, SMS, and a few other types of campaigns.
On the other hand, the default ERPNext CRM is extremely basic. It only contains an email type of campaign. This is even more of a problem because Thank U Foods uses telemarketing and SMS, not email. And then they don’t have the concept of customer retention — you could only perform campaigns on leads. And even in that, a lead could only have one campaign.
Moreover, we had to associate existing leads manually(new leads could be added by a button which they had in their campaign detail page). That means that if we wanted to add an existing lead, we had to go to that lead, open the detail page, scroll down, and then select the source as a campaign, and then select the campaign. In Salesforce, there is a tool that lets us bulk-associate members by filtering the members to import based on the fields.
What’s more, the way to associate existing leads isn’t mentioned in the docs. The first time we used the CRM, we thought that we couldn’t associate existing leads with a campaign. Only after playing around did we realize that we could fill in this field.
Then, we needed to import the customers from Shopify, which is what we use for our e-commerce site. The default data import, while being quite good, didn’t fit our needs because of the ERPNext data modeling. The Shopify export Excel contained all the contact details (name, phone, email, company) and the address details (city, state, country, etc.) in the same file. But in the Lead model, the name is stored as a field in the Lead, the other contact details are stored as fields in an FK to Contact (which is not the Salesforce contact, here the Contact model represents a person), and the address details are stored in an FK to Address. And the data import does not create foreign keys.
In summary, we couldn’t use the default CRM at all. It didn’t fit any of our needs. So we had a few things to do:
- Remodel the data to associate both leads and customers with multiple campaigns.
- Implement a way to bulk-associate leads and customers with campaigns.
- Write a data import that created and associated addresses, contacts, and, if possible, campaigns along with a member. In the future, write hooks that automatically imported Shopify data into ERPNext.
- Add support for SMS, telemarketing, and postal campaigns.
DB Schema
Although in theory the easiest, the “remodel the data” step turned out to take the most amount of time. With no knowledge of databases, schemas, and their likes, I’d estimate that around half of the time spent was spent on this. Over the weeks, we kept coming back to how the DB should be structured, without any conclusive schema.
We first wanted to compare both the Salesforce schema and the ERPNext one by drawing ER diagrams. We could access the Salesforce schema on the Salesforce portal. For the ERPNext one, I installed DBeaver(brew install — cask dbeaver-community) and loaded my local MariaDB database into it.
To my shock, I found out that Frappe doesn’t use FKs in its backend. Frappe CEO Rushabh Mehta explains in this post why. But as a result, we couldn’t draw ER diagrams. Which meant that we couldn’t compare both. At the end, we manually drew ER diagrams.
What we wanted was a new Campaign Member DocType (Frappe’s fancy term for a model) which had the fields of campaign, status, and member (customer or lead). This was possible through the use of dynamic links, a Frappe field type that allowed you to link to an instance of more than one DocType, as opposed to a normal FK where you can only link to an instance of a DocType. So by setting the member to be a dynamic link of Lead and Customer, I could link to either in one field.
But creating this new DocType led to lots of problems. First, the reports became useless, because the reports get leads of a campaign by fetching all the leads with source=campaign and campaign_name=this_campaign. But with our schema, the reports needed to fetch all the campaign members by seeing which campaign members had campaign=this_campaign. So, we would either have to rewrite them, or not use them at all.
Also, a lot of fields became unnecessary. Some of them were definitely unnecessary — there was obviously no use for the source and campaign name fields anymore, for example. A lot of other fields seemed unnecessary, and we first considered removing them to save DB space, but then, on my father’s advice, didn’t. It could break functionality somewhere else, especially with the Customer DocType, which was associated with the retail section, not the CRM section.
Finally, we repeatedly reconsidered the structure based on some new requirement, problem, or advice. For example, I had originally added a contact field to Lead, and made a few fields like first name read-only (I also hid them and made them default to contact.field_name ), so that I could update the contact details only from a lead’s contact. But this led to the problem that in the front end, I couldn’t access the contact’s details. I couldn’t do doc.contact.first_name while filtering in reports, or when searching in lists. So, I rolled back all of the changes, which was painful and time-consuming work. I had to go to every field and untick the hidden and read-only checkboxes, and clear the “Fetch From” box. What’s more, I did this whole routine twice.
We got into contact with two consultants/developers to help us. Both of them gave opposing views about the DB structure. The first agreed that we customize fully, the second advised us to go with the flow (not change anything except the crucially crucial, and go with the defaults). Because we met the second one in the fourth week, when we had just got some idea about the schema, this advice meant that we rollback all the work of a month. We decided not too.
Frappe
The main reason behind us terminating the project was Frappe Framework itself (without ERPNext). It had lots of cons, and too less pros to balance out. Most importantly, it was slow. Too, too, slow. Deleting a file takes 5 seconds. 12 files would many a times just hang, and reloading the page stopped the deletion, which meant that clearing the file system could take up to 10 minutes. (Edit: I repeat, this should have changed in the recent update)
A page takes 10–20 seconds to load — with cache. Creating a new site takes minutes. After you create, you’ve got to install ERPNext, which takes a few more minutes. Then, you’ve got to run something called a set up, which takes even more time.
This isn’t very surprising considering the size of the framework. The ERPNext app folder is 1.2 GB and the Frappe Framework folder half a GB. The whole project folder — what they call the “bench” — with minimal customization, is 2.3 GB. They have 1000+ DocTypes in the DB as soon as the site is created. It’s basically huge in all aspects.
Then the docs are very poor. It doesn’t list many methods which are in the framework (for example, there’s no mention of editing the DB using JS, which is actually there). In many places, it’s ambiguous(frappe.require is, it seems, used to load libraries that are not used “often” — rather vague). In a few places, the official documentation is outdated. Following their instructions, I spent a week trying to install and get my site up and running.
Lastly, the framework’s undeveloped. The ORM’s really basic. They are like two representations of a DocType — a dictionary with the fields, which doesn’t have any of the special methods of that DocType, and the actual DocType class. Some functions return this, some return that. You can’t get a list of the actual documents — you can’t, say, get a bunch of files and read them, because the File.get_content function is only on the actual DocType, and the function to get a list of instances returns a list of dictionaries.
In many of the DB functions, you can’t pass filters, only the PK. So if you want to read the file with the file name of customer_dump.csv, you’ve got to have two (rather ugly) lines of code:
file_dict = frappe.db.get_list("File", filters={"file_name": "customer_dump.csv"})[0]
# name is the PK field - `get_doc` only takes the PK
file_contents = frappe.get_doc("File",
file_dict.name).get_content()Also, it’s buggy in multiple instances. Here’s just a few listed:
- As soon as a site is created, it gives an error about Administrator settings already set.
- In a few pages the home page link doesn’t work — raises a 404.
- Updating some of the list settings (while viewing all the instances of a DocType) doesn’t update the UI immediately, you have to refresh the page.
- Frappe is a Single Page Application. Which is OK if it’s fast. It isn’t, so when you click a link, it often seems to hang.
- In the sidebar navigation, you can’t see the current page in the navigation list, you have to scroll down the list and find it.
- Some of the macOS keyboard shortcuts conflict with other shortcuts. ⌘ + H is supposed to navigate to the home page, instead it hides the application. Shift+⌘+R, to clear cache and reload, instead opens up the Reader in Safari.
Testing
Writing tests was extremely difficult. Because Frappe is an SPA, it stored the previous pages’ HTML in the DOM. So when I had a Cypress selector cy.contains(“Add Campaign”).click() to click the Add Campaign button, it instead clicks the Add Campaign Member button, which exists in the previously opened page. Labels are not associated with inputs, and most inputs don’t have placeholders, which makes it almost impossible to select an input without using a CSS selector. On rereading their page on testing, I noticed they suggested using Testing Library. While certainly helpful in selecting buttons, it still didn’t work with inputs. Timeout errors where common, too, because of the how much time some of the pages take to load.
Things I liked about Frappe/ERPNext
While there were a lot of faults, as mentioned above, their are certainly some good things about Frappe. For example, the Frappe Desk is the Admin panel. It’s incredible. Its styling is a lot better than, say, Django’s admin panel. It has a lot more features, too. By connecting an email account, you can send emails to people from the Desk. It tracks front-end changes made to a document (an instance of a model).
The list view of a model is really modern, too. You can search, filter, and even save filters (a very nice feature). It automatically refreshes the list as new documents are created. You can change the columns listed in the frontend itself. Other than a normal list view, you have a report, dashboard, calendar, Kanban, Gantt, and a tree view.
The Awesomebar, which is used to search all the pages in the whole desk, really is awesome. From anywhere, you can go anywhere. None of the link clicking which you normally have to do. Just a search, and you can go to that page.
The collaboration features are really good, too. They have chat rooms, a way to schedule events, a ToDo DocType, and a notification centre. You can assign DocTypes to people. You can track activity online, give bounty points based on usage, and give users a wide variety of permissions. And I’m only digging the surface, I haven’t actually used Frappe in collaboration, and I’m sure there are a lot more great features.
From the desk, you can generate a pretty good website for the frontend. You can add pages, have forms, and even write blog posts. You can automatically generate docs for your site or app, too.
Abrupt end
And… yeah, that’s all I wrote last year, and it kinda suddenly jerks to an end. I hope this has helped you understand ERPNext a little better, and please do share you thoughts in the comments if you use it!
