‘Spark is becoming more important than Hadoop’

by | Mar 8, 2016

Spark can run in the Hadoop ecosystem, or it can run in its own stand-alone environment. Over 25% of Spark projects today run outside of Hadoop, and the percentage is rising. Moshe Kranc, Chief Technology Officer, Ness Software Engineering Services (SES) talks about the big data trends for 2016 and Spark’s role over Hadoop. The views and opinions expressed in this article are entirely those of Moshe Kranc.

— Vinita Gupta Malu

Q. What are the Big Data trends that you observe in 2016?

Moshe: I’ve observed the following trends:

  • Spark is becoming more important than Hadoop: Hadoop has been around since the late ‘90’s, and has evolved to the point where it can efficiently and reliably perform big data analytics. Spark has the advantage of being a fast follower, able to learn from and avoid Hadoop’s mistakes. Spark has a more generic and extensible programming model, which makes it easier to use for analytics. It also can handle big data in Motion, via Spark Streaming, and serves as the basis for a powerful graph database (GraphX) and a full-featured data science library (MLib). Spark’s closest relative in the Hadoop world is Tez, which, like Spark, can execute algorithms organized as directed acyclic graphs. The open source community, recognizing the similarity, has crowned Spark as the converged platform of choice, and it will soon replace Tez in the Hadoop platform. Spark is the future of big data computing.
  • There will be fewer big data startups:  Venture capital investors view big data as last year’s trend. They have already doubled down on a variety of startups, and want to see those investments pan out. Hence, this year it would be difficult for the big data startups to convince the investors.
  • Oracle continues to lose market share to open source big data technologies: The mainstream software giants have adopted various strategies to cope with the competition from open source big data platforms. Some have formed alliances (e.g., Microsoft and HortonWorks), some have embraced and extended (e.g., IBM Watson). The company that least seems to get this brave new world is Oracle, which continues to sell Exadata (an expensive alternative for big data analytics), and has launched their own proprietary NOSQL database that has no advantage over open source alternatives. Oracle is having trouble understanding that in 2016, most customers prefer to avoid vendor lock-in.
  • Cassandra is becoming a dominant player in the NOSQL space: Cassandra was always the fastest NOSQL database, especially for write-heavy applications, and it provides an active-active distributed datacenter topology out of the box. The knock on Cassandra was that it was hard to deploy, maintain and program. Datastax, the commercial vendor for Cassandra, seems to have noticed: The CQL language makes Cassandra far easier to program, and the OpsCenter management tool makes maintenance a lot simpler.


Q. What according to you would be the big data challenges in 2016? Explain the ways to tackle it.

Moshe Kranc, CTO, Ness Software Engineering ServicesMoshe: If there is one common thread that links 95% of Ness’s big data customers, it is FUD – fear, uncertainty, doubt. The field is full of competing or overlapping products, each of which claims to be the Holy Grail for big data. The big boys (Oracle, IBM, SAP, Terradata, etc.) all want to steer you towards their (costly) offering. The upstarts (Cloudera, HortonWorks, DataBricks, DataStax, etc.) all talk about use cases with dramatic savings, but do not tell about use cases where their product fails miserably. Depending on the vertical, there are dozens of one stop shops who offer to take the data and come back with vertical-specific insights.

Left to evaluate this cacophony of conflicting voices is your organization, with little experience in the big data minefield. No wonder we read so often about big data failures. The real culprit is not Hadoop – the real culprit is the silo approach that drove the company to make a choice without the benefit of outside advice or experience. The only way to cut through the hype around a product is to try it yourself, and/or talk to someone you trust who is using it. Another option is to partner with a company like Ness Software Engineering Services that has seen a broad range of big data projects and technologies, and has a proven track record of success.

Q. Will spark overtake Hadoop?

Moshe: Hadoop as a concept revolutionized the world of data processing, and ushered in the era of big data. But, Hadoop as a product ecosystem is certainly showing its age, and, for many use cases, it has been upstaged by more modern technologies. Therefore, Hadoop is not necessarily the safe choice for your big data use case. In the long run, Hadoop may lose out to newer products like Spark and Cassandra, which had the benefit of learning from Hadoop’s growing pains.

Q. What are the best practices for migrating a project from Hadoop to Spark?

  • Do the migration in stages.
  • Get your code working on Spark as quickly as possible: If you are using Hive, find the transition to Spark SQL fairly painless. If you are using MapReduce thenoften reuse your mapper and reducer functions and just call them in Spark, from Java or Scala.
  • Once your code is running on Spark, tune the performance, and consider re-coding your algorithm to take advantage of Spark’s ability to execute algorithms structured as Directed Acyclical Graphs.


The views and opinions expressed in this article are entirely those of Moshe Kranc, Chief Technology Officer, Ness Software Engineering Services.


Share This Article!

Brian Pereira
Brian Pereira
Brian Pereira is an Indian journalist and editor based in Mumbai. He founded Digital Creed in 2015. A technology buff, former computer instructor, and software developer, Brian has 29 years of journalism experience (since 1994). Brian is the former Editor of CHIP India, InformationWeek India and CISO Mag. He has served India's leading newspaper groups: The Times of India and The Indian Express. Presently, he serves the Information Security Media Group, as Sr. Director, Editorial. You'll find his most current work on CIO Inc. During his career he wrote (and continues to write) 5000+ technology articles. He conducted more than 450 industry interviews. Brian writes on aviation, drones, cybersecurity, tech startups, cloud, data center, AI/ML/Gen AI, IoT, Blockchain etc. He achieved certifications from the EC-Council (Certified Secure Computer User) and from IBM (Basics of Cloud Computing). Apart from those, he has successfully completed many courses on Content Marketing and Business Writing. He recently achieved a Certificate in Cybersecurity (CC) from the international certification body ISC2. Follow Brian on Twitter (@creed_digital) and LinkedIn. Email Brian at: [email protected]
Recommended Posts
Why Landing On The Moon Is Difficult

Landing an uncrewed vehicle on the surface of the moon or any planet is not easy; space institutions have made multiple attempts after seeing their spacecraft crash on the surface.

Similar Articles

Share This