Analytics has become a man power intensive activity.Man power is very expensive on top of it Analytics field is replete with jargon, as a result it has going beyond the reach of smaller companies. With today’s advances in technology there is no reason why a large part analytics can be automated. So taking cue from “lean manufacturing” and “lean startup” we should also aim towards “lean analytics”.
A large part of lean analytics would involve automating various steps, reducing wastage of analysis and data, and building compute-efficient models. It all starts at data gathering stage. Once the data is machine readable most of the other analytics can be automated.
There are ready tools for data munging, ETL etc. Statistical analysis can be completely automated. Even model building and prediction can be largely automated. So data gathering is the key for making analytics lean. So here are the hygiene factors for lean data collection:
- Reduce human error as much as possible through data collection automation (for ex. have drop boxes in forms instead of text)
- Try to figure out data model in advance but keep scope for unstructured data
- Dont collect all the data you can get. You have something doesn’t mean you need it
- Set a process for cleaning before storing
Automation is the biggest growth driver for any industry. Much work in being done in automation of various aspects of business: work flows, project management, marketing etc etc.. Business Analytics is lagging quite behind in this though. Much of business analytics today involves manual data munging, analysis by an expert, modeling through the use of statistical tools and a final decision making by a manger. Bulk of this process can be automated. Imagine a system that looks at data, converts data into usable format, identifies trends, presents those trends in human readable reports and generates actionable items that can be read by other systems. That is where analytics should head to.
This vision will involve coming together of several fields of science. There will automated statistical models identifying trends. There will be business expertise converted into machine readable rules. There will be natural language generation and interpretation. There will be smart dash-boarding. And of course there will be Databases, APIs and UIs All of this together will be what I call artificial intelligence in business analytics. Parts of it is already happening when an e-commerce company decides to give a customer discount on a particular product on the fly or when an algorithm decides to do a particular trade in financial markets. In fact eventually it can lead to complete automated decision making in all aspects of business! The only control remaining with humans being the plug to the machine.
Banks are entities with several disparate services. This has been for legacy reasons as people trusted banks for all money related business (except may be insurance). thus banks offer payments, deposits, loans, wealth and asset management, broking, custodial , transaction advisory, investment banking and other varied set of services. This variety could have also been necessary when economies of scale are not achievable in one single line of business. This is no longer necessary in the current scenario when economies of scale can be achieved through technology. Thus there is a scope for breaking the bank into several smaller entities serving specific need of the customer. This is what is exactly happening right now in the fintech world. And this is how the bank is getting unbundled:
- Digital wallets and gateways are replacing payment solutions
- Online liquid funds are replacing deposits (especially in China. Not yet happening in India)
- P2P lending and automated lending are replacing traditional bank loans
- Robo-advisors are taking over traditional wealth management
- Penny brokers are replacing broking services. this revolution has to still disrupt OTC broking though
- Several crowd sourcing platforms are replacing investment banking at least for smaller firms
Some areas where fintech has still not made a significant mark is:
- Smart beta investment products at very low cost
- Custodial and security services are still primarily handled by banks as lot of trust is involved
- International payments because of regulatory issues
The only way for a bank to beat the onset of fintech revolution is to be part of it. This involves several things:
- Upgrading technology to latest stacks with more open architecture and leveraging services of other tech players
- Digitising the offerings
- Using data analytics for increasing customer LTV and reducing inefficiencies
Looking to build a BigData platform? Here is an infrastructural checklist that you should build:
- Scalable structured DB (PotsgreSQL is a good choice)
- Scalable unstructured DB (MongoDB)
- Streaming data (Apache Storm)
- In-memory data processing (Apache SparkSQL and RDD)
- Machine learning (Spark MLLIB)
Also Python and associated libraries: although its not a big-data tool in itself python is a great language with readily available packages to interact with most big-data tools.
Many proponents of NoSQL think that tables are so yester-century. Even in HTML Divs have replaced tables long time back for placing content on the page. Does that mean SQL will die down? May be, may be not..
Firstly there are several companies invested heavily in structured data solutions. Replacing all of them will take considerable amount of time. On top of it NoSQL solutions do not have standardisation to compete with the existing SQL-like frameworks. Also some data is inherently structured and tabular in nature always. So a combination of structured and unstructured is what is required. PostgreSQL handles this using tags. It is good but not enough for making it a standard. MongoDB rules the unstructured DB world at the moment but it is poor when it comes to handling data frames. So still waiting for a good “Not-only” SQL DB.
My wishlist for this is:
- Ability to handle key-value pairs and also data frames.
- A good query language (preferable not too different from SQL)
- Great documentation and support for generic programming languages (Java, R, PHP, Ruby atleast to start with)
MongoDB with inbuilt dataframe objects and a query language to access these frames could satisfy most of these items. Hopefully they are listening and will do something soon