ELK Stack

• ELK is also called as elastic stack.
• Elastic stack is a group of open source products built by elastic.
• It has a set of different tools used to connect, analyse, and visualise data.
• Elastic stack refers to a set of open source products that have been developed by elastic to help its users collect data from different types of sources and then analyse the collected data and represent it in an easy to understand and aesthetic visualisation. This is done so that meaningful observations can be made.
• For example, in a code pipeline, how request are coming to code pipeline, How well it is working, time it is taking at different steps, Like, build, code, analysis, Deployment, et cetera.
• ELK is highly scalable.
• When we expand our product across multiple servers we can analyse health of all those servers.
• ELK does all the processing in real time.
• It can be attached to an application working in current time that is while it is being used by customers.
• All the logs of people accessing application, what they are accessing, where they are accessing from, what browser they are using as stored and analysed and processed by ELK.
• ELK then prepares bespoke visualisations for analysis by end user.
• ELK is an open source software.
• ELK comprises of following components.
• Elastic Search
• Elastic Search is a distributed no sql database which uses json like messages where you can store messages in the form of json across distributed systems where you can scale across indexes.
• Elastic search is a search engine based on Apache leucine library.
• Elastic search is a no SQL database, which can be communicated through json request.
• Elastic search provides near real time, search and analytics for all types of data. Whether you have structured or unstructured text, numerical data, or geospatial data. elastic search can efficiently store and index it in a way that Support fast searches.
• Used for storing and Searching collected data.
• Used for log analytics, full text search, Security, intelligence, business analytics, and operational intelligence.
• Elastic Search is a distributed system you can expose restful API’s.
• Elastic search is a no sql database That was developed Based on Apache Lucene search engine.
• Data is stored as a document. Document data is then separated into fields.
• We quay data using Rest API’s.
• It provides a distributed full text, search engine with an HTTP web interface.
• It is schema free json Documents.
• Elastic search is developed in Java and is released later as open source under the terms of Apache license.
• It can be used To index and store multiple different types of documents and data.
• It provides a function to search the data that is stored in real time as it is being fed.
• Need of elastic search
• Elastic search is able to achieve fast search response because instead of searching the text directly, it searches on index.
• Components of elastic search
• Index
• An index consist of one or more documents
• Document
• Document consist of one or more fields
• Shards
• Split up, indices horizontally into pieces called shards.
• Replica
• One more copy of index.
• Log Stash
• Log stash connects data from different sources and streams the data in the form of data processing pipeline on the elastic search instance.
• Elastic search is the data holder and log stash streams data to the holder.
• Used for collecting and filtering the input data.
• Send to elastic search for storing and indexing.
• Log stash is a collection agent and is used to collect both heterogeneous and homogeneous data from various sources.
• It has the capability to screen, breakdown and make string alteration in the data it collects.
• After it has collected and filtered the data it then send it to elastic search for storage.
• Kibana
• Kibana is the UI Component which displays data from elastic search instance.
• Provides a graphical user interface
• We can create bespoke visualisation.
• Kibana is a graphical User interface that is used to display the data that was collected and stored in elastic search.
• It displays them With appealing visuals, so that the data could be easily understood and analysed, it does so using multiple different types of visuals, like bar charts, pie charts, world maps, heat maps, coordinate maps, et cetera.
• Discover the data exploring it.
• Analyse the data by applying different metrics.
• Visualise the data by creating different types of charts.
• Apply machine learning on data to get data anomaly.
• Manage users and roles.
• A console to run elastic search expressions.
• Play with time, series data using timeline.
• Monitor elastic stack using monitoring.
• ELK stack further adds One more component To the ELK
• Beats
• Multiple lightweight data collectors.
• Does not filter data like log stash.
• Collect data much easily, since it has lightweight small tools.
• These small tools attach them to sources to collect data.
• Beats is similar to log stash In matter of fact that they both collect the data that will be later stored and analysed, but beats differs in the method of collection.
• Beats are multiple small software installed on different servers From where they collect data and send it to elastic search.
• Different types of beats are
• File Beat
• They can directly send the data to the elastic search, or they could send it to log stash Where data can be filtered using log stash.
• Used to collect information on logs or log file.
• File beat can be used for more efficient processing of real time data.
• File Beat Is more automated, then logstash.
• File beat does not require a pipeline to be written.
• Check for enabled file beat modules and enable nginx and system.
• Metric Beat
• Gatherers metrics Like RAM, CPU USAGE, Input/Output, Application Runtime To check if application is running or not.
• Packet Beat
• Used for checking analysis on networking data, helps in figuring out discrepancies in data that may be trafficked on the network.
• Actual data being sent and received.
• Help to figure out any attacks or misuse of Network.
• Audit Beat
• Used for collecting Audit events which contains details like unauthorised access or any events based on a criteria set.
• Winlog Beat
• Works on windows systems keeps track of events like hardware, security, and applications on Windows based systems.
• Heartbeat
• Checks for uptime, records, pink statistics, check if remote server is reachable or not.
• Beats was later added to ELK and ELK name was changed to ELK stack.
• ELK Flow
• How data flows from one software tool to another in ELK stack.
• First, the beats are attached to remote servers from where the beats collect information from various sources.
• After collecting all the data needed, they either ship the data to log stash for filtration or directly send the data to elastic search.
• The data is getting stored in elastic search from here. It will be not be directly sent to Kibana. 
• Kibana First needs to check where elastic search is, and then go and get the data itself.
• Features of ELK
• System performance monitoring
• Log management
• Application performance monitoring
• Application data analysis
• Security monitoring and alerting
• Data visualisation
• Using Kibana
• Collect static Apache logs using log stash And Analyse them using Kibana
• Collect static ‘.csv’ using Logstash and analyse them using Kibana.
• Collect real time, weblogs, and configure, beats to inject them into elastic, search and analyse them using Kibana.
• To visualise and explore data in Kibana we must create an index pattern in logstash And map it to a view In Kibana.
• An index pattern tells Kibana Which elastic search Indicis contain the data we want to work with.
• An index pattern can match a single index, multiple indices And a roll of index.
• Index pattern indexes The concerned data in logstash. It is like categorising the data that you want to see.
• We see the keywords related to index on Kibana dashboard.
• We can also apply filters to the data.
• We can use Kibana Query Language Which offers A simplified query syntax and support for scripted fields.
• We can change the time period of logs.
• Configure ELK from as described in https://springimplant.blogspot.com/2024/04/elk-stack.html