Are there some resources I can use to resolve/delete things creating high server load?

Hello,

I have recently overloaded one of my Squirro instances with too much articles and machine learning jobs. This means that I am reaching the memory limit of my instance with the level of resources I have on the server. I’m wondering what is the best way to clear up some of these things. First of all, how can I check the load across various squirro services to isolate machine learning v. elastic search. Then, second how can I clear up those items on the server? I imagine there must be some documentation around this, so any help to point to the right direction to documentation or advice would be great.

Any cleanup scripts would be a bonus. Thanks!

Best,

Manu Mulaveesala

2 Likes

Hey @manutej thanks as always for your questions.

You can monitor your Squirro Server through the UI by clicking on the Squirro Spaces (grid icon in the top right-hand corner of the screen) > Server > Monitoring. This will give you some stats on disk and memory usage.

If you have access to the command line you can login to your server and use the top command to better understand how the CPU is using the server resources.

You can clean up any unwanted machine learning jobs via the UI by going to the AI Studio tab > ML Jobs and remove the jobs manually.

Alternatively, if you have access to the command line I can point you to documentation on Managing the Squirro Pipeline and Managing Elasticsearch.

1 Like

A common issue is that Elasticsearch is allocated with too much RAM when it’s running on the same server/instance as the Squirro application.

So you might benefit from lowering the RAM allocation of Elasticsearch. It’s not a Memory database, so not all data has to fit into RAM. RAM for ES becomes more critical with huge data sets or when you have lots of concurrent users. So in DEV environments that is seldom the case.

Check out this guide on how to adjust the memory setting of Elasticsearch https://squirro.atlassian.net/wiki/spaces/DOC/pages/255295524/Managing+Elasticsearch#Memory-setting

Another issue could be that you allow allow multiple machinelearning processes on a machine that is low on resources. Here it would be good to have only one process, instead of two or three fighting for resources.

You can see this also in top, if there are more than one sqml and machinelearning process running.
To control it you can edit the file /etc/squirro/machinelearning.ini

Ensure this is added/updated accordingly:

[server]
fork = true
min_spare =1
max_spare = 1

Hope this helps,
Toni

1 Like