Cloud is a bigger revolution than mobile

Much of the boom in the tech space is attributed to rapid increase in connectivity through mobile. That is a tremendous underestimation of what is going on right now in the tech world. Most of the ease if setting up and running businesses is because of the evolution of cloud architecture and not just because a user is able to access an app on phone.

Although mobile connectivity is serving the last mile, a significant amount of processing happens before the service reaches to the end user. And all of this has become seamless because of the cheap availability of compute, storage and bandwidth on the cloud. This much ignored fact leads to some misjudgments on several business models. Firstly if a business is serving only the end user without fully leveraging the power of cloud will never be efficient and will lose out to the competition soon enough. On the other hand, businesses purely focused on delivering cloud based services either directly to the users or to other applications will be definitely adding real value to the ecosystem.

In other words for any tech oriented company: “if you are not on cloud, you are dead”. Being on the cloud does not mean putting up a server on AWS or Azure. It involves 1. producing services for internal and external consumption, 2. consuming cloud services where ever possible rather than reinventing the wheel and of course 3. hosting data and compute on the cloud. In other words you have to be part of the cloud ecosystem.

Big Data to Lean Data

Analytics has become a man power intensive activity.Man power is very expensive on top of it Analytics field is replete with jargon, as a result it has going beyond the reach of smaller companies. With today’s advances in technology there is no reason why a large part analytics can be automated. So taking cue from “lean manufacturing” and “lean startup” we should also aim towards “lean analytics”.

A large part of lean analytics would involve automating various steps, reducing wastage of analysis and data, and building compute-efficient models. It all starts at data gathering stage. Once the data is machine readable most of the other analytics can be automated.

There are ready tools for data munging, ETL etc. Statistical analysis can be completely automated. Even model building and prediction can be largely automated. So data gathering is the key for making analytics lean. So here are the hygiene factors for lean data collection:

  • Reduce human error as much as possible through data collection automation (for ex. have drop boxes in forms instead of text)
  • Try to figure out data model in advance but keep scope for unstructured data
  • Dont collect all the data you can get. You have something doesn’t mean you need it
  • Set a process for cleaning before storing

 

 

 

AI in Business Analytics

Automation is the biggest growth driver for any industry. Much work in being done in automation of various aspects of business: work flows, project management, marketing etc etc.. Business Analytics is lagging quite behind in this though. Much of business analytics today involves manual data munging, analysis by an expert, modeling through the use of statistical tools and a final decision making by a manger. Bulk of this process can be automated. Imagine a system that looks at data, converts data into usable format, identifies trends, presents those trends in human readable reports and generates actionable items that can be read by other systems. That is where analytics should head to.

This vision will involve coming together of several fields of science. There will automated statistical models identifying trends. There will be business expertise converted into machine readable rules. There will be natural language generation and interpretation. There will be smart dash-boarding. And of course there will be Databases, APIs and UIs:) All of this together will be what I call artificial intelligence in business analytics. Parts of it is already happening when an e-commerce company decides to give a customer discount on a particular product on the fly or when an algorithm decides to do a particular trade in financial markets. In fact eventually it can lead to complete automated decision making in all aspects of business! The only control remaining with humans being the plug to the machine.

Fintech Revolution aka Unbundling of the Bank

Banks are entities with several disparate services. This has been for legacy reasons as people trusted banks for all money related business (except may be insurance). thus banks offer payments, deposits, loans, wealth and asset management, broking, custodial , transaction advisory, investment banking and other varied set of services. This variety could have also been necessary when economies of scale are not achievable in one single line of business. This is no longer necessary in the current scenario when economies of scale can be achieved through technology. Thus there is a scope for breaking the bank into several smaller entities serving specific need of the customer. This is what is exactly happening right now in the fintech world. And this is how the bank is getting unbundled:

  • Digital wallets and gateways are replacing payment solutions
  • Online liquid funds are replacing deposits (especially in China. Not yet happening in  India)
  • P2P lending and automated lending are replacing traditional bank loans
  • Robo-advisors are taking over traditional wealth management
  • Penny brokers are replacing broking services. this revolution has to still disrupt OTC broking though
  • Several crowd sourcing platforms are replacing investment banking at least for smaller firms

Some areas where fintech has still not made a significant mark is:

  • Smart beta investment products at very low cost
  • Custodial and security services are still primarily handled by banks as lot of trust is involved
  • International payments because of regulatory issues

The only way for a bank to beat the onset of fintech revolution is to be part of it. This involves several things:

  1. Upgrading technology to latest stacks with more open architecture and leveraging services of other tech players
  2. Digitising the offerings
  3. Using data analytics for increasing customer LTV and reducing inefficiencies

Checklist for BigData

Looking to build a BigData platform? Here is an infrastructural checklist that you should build:

  • Scalable structured DB (PotsgreSQL is a good choice)
  • Scalable unstructured DB (MongoDB)
  • Streaming data (Apache Storm)
  • In-memory data processing (Apache SparkSQL and RDD)
  • Machine learning (Spark MLLIB)

Also Python and associated libraries: although its not a big-data tool in itself python is a great language with readily available packages to interact with most big-data tools.

SQL.. NoSQL..WhateverQL..

Many proponents of NoSQL think that tables are so yester-century. Even in HTML Divs have replaced tables long time back for placing content on the page. Does that mean SQL will die down? May be, may be not..

Firstly there are several companies invested heavily in structured data solutions. Replacing all of them will take considerable amount of time. On top of it NoSQL solutions do not have standardisation to compete with the existing SQL-like frameworks. Also some data is inherently structured and tabular in nature always. So a combination of structured and unstructured is what is required. PostgreSQL handles this using tags. It is good but not enough for making it a standard. MongoDB rules the unstructured DB world at the moment but it is poor when it comes to handling data frames. So still waiting for a good “Not-only” SQL DB.

My wishlist for this is:

  • Ability to handle key-value pairs and also data frames.
  • A good query language (preferable not too different from SQL)
  • Great documentation and support for generic programming languages (Java, R, PHP, Ruby atleast to start with)

MongoDB with inbuilt dataframe objects and a query language to access these frames could satisfy most of these items. Hopefully they are listening and will do something soon:)

Models in Financial Markets

Prediction is a key component of financial market modeling. In fact its an area where prediction can be applied directly. Risk/Return optimization is the other important aspect in financial market modeling but that is secondary to prediction. The third important aspect is pricing especially of derivative products with non-linear payoffs. This third part is more or less solved and there are standard practices here.

So it boils down to prediction and optimisation when it comes to modeling in markets. And this is a very fertile area. Prediction in financial markets especially is very interesting because of the null hypothesis of efficiency of markets. In other words all information is already priced in. This induces certain discipline in predictive modeling.  Also if there is a model that is able to predict better than the market, eventually market forces will make the model less and less accurate. Several non-finance professionals be it analytics professionals, statisticians or machine learning professionals do not completely appreciate the gravity of this hypothesis. Thus they get carried away by initial success of a particular model that might have worked in a particular regime. This is where strong understanding of financial principals are needed for building a robust predictive model and the model itself requires continuous monitoring and updating. Thus there will never be a dearth of demand for quants in financial markets.

Optimisation on risk-reward is actually an easier part but still many people (even in finance) do not appreciate the power of this. Here particular finance knowledge will not add significant value except that having some understanding of CAPM and other models will make optimisation a much simpler task.

The last bit of pricing models is one are where quants have pored significant brain power and it has come to a stage where the models are stable and can be taken off the shelf. Now a days people no longer build new option pricing models instead they have moved the focus to better prediction of volatility. Which is a good thing cause the problem has moved from the world of complex stochastic calculus to a simpler world of statistical analysis. Nevertheless there is work to be done in building computationally efficient models. For example some problem statements are: how to parallelize a path-dependent simulation? how to reduce number of simulations required drastically? etc.. This is a place where core quants and computer scientists can come together.

To conclude there is a strong need for data analysts, quants and statisticians in financial markets but they need to add on some financial skills before jumping on to modeling.

 

 

 

Follow

Get every new post delivered to your Inbox.