News & Updates

Fix for Full SD Cards

Hi everyone! We recently had a bunch of hotspots that got full SD cards and weren’t cleaned automatically, which is usually the case. We have talked to the Helium developer team now to know how to mitigate these cases.

We have now released a script to every hotspot that will automatically clean up the SD card (like Blockchain cache and more) when it is close to going full. The script works as follows:

• Check if the SD card is more than 80% full (check happens once an hour)
• If yes, initiate a cleanup of the miner data folder
• Restart the miner software

As soon as the miner software restarts, it will take roughly 30-60 minutes until it is “useful” again. From then on, you can check your dashboard and see what the current miner height is. If it says “1” or “-1”, don’t be worried, it just means it needs a bit more time.

After roughly 3-4 hours, please check again, and if you still see low height, please issue an instasync through the instasync button.

All miners already received an update, however we have a handful of them that couldn’t get the emergency commands through. We are fixing them one by one at this moment now. Fixing should be finished in roughly two hours.

We will closely monitor the script on every hotspot and have implemented an alarm system through our management dashboard for hotspots that go over the 80% limit, so that manual intervention can happen before disaster strikes.

Info on Firmware 2.0 and Dashboard 1.06

Deployment of Firmware 2.0 is starting very soon (6PM UTC). Just a reminder to keep your devices connected to Ethernet tonight, thank you! Here’s some information on this release.

Bug Fixes for Device Firmware 2.0

• “Wrong Wallet Adress” error for QR Code onboarding: This error is now fixed and wallet adress can be read correctly. As always, let the device run for at least 10 minutes until generating the QR Code, as the miner software is very busy when first booting up.
• Dashboard stays or goes offline: This error is now fixed as well. There’s now a service running that automatically checks the state of the dashboard and reliably re-enables it after reboot. After tests on 50 devices that had this behaviour, the dashboard is running perfectly fine on all 50 devices after multiple days and forceful reboots.
• Miner stuck at specific sync height: On some devices, the miner stopped syncing, but was otherwise working normally. During installation of Firmware 2.0, the miner will check if sync_state of the miner container is active. If not, it will re-enable it, which will resume syncing. If it doesn’t work, it will do several checks and do an automatic re-installation of the container if needed. Furthermore, a button to resume sync if not active will be added to the next Diagnostics Dashboard version 1.07
• Packet Forwarder not running: This is a very rare issue, but still happened on a handful of devices. We have applied a monitoring script that checks if the LoRa packet forwarder is running. If not, it will do several checks and attempt to automatically self-heal itself. The miner container will be stopped for a few seconds during this process and re-enabled after everything works fine.

Diagnostics Dashboard 1.06 Updates

The 1.06 Update was already enabled on most of devices, and for those who didn’t got one, it will be shipped with Firmware 2.0. These are the improvements:

• Easier Onboarding with Deep Link: You can now press a button on the dashboard instead of scanning a QR code. This makes onboarding more easy if you logged into it over the phone. Just press the button beneath the QR code after generation. The button contains a deep link to the Helium app – the app should start up automatically and ask you to register the device.
• Instasync re-enabled: Instasync is now re-enabled after multiple issues were identified. Instasync will grab a snapshot of the Blockchain that is max. 240 blocks behind. Instasync command takes quite a while (up to 10 minutes), so please don’t press anything in the dashboard after you have started instasync and wait until the command finishes. Also, don’t use instasync if you are already synced (roughly 200-250 blocks behind the Helium blockchain max).
• Several minor bug-fixes
  

The much requested password change feature is on our high priority list for this week, as well as more granular diagnostics options.

Firmware 2.0 Update today

Hi everyone! We are now doing our last tests before deployment of Firmware 2.0 – please keep your miners plugged in and connected to Ethernet today, update is planned in the time between 6PM and 11PM UTC today.

Firmware 2.0, as already mentioned, is fixing lots of the errors some users had in the last weeks, and incorporates monitoring and self-healing scripts if something should go wrong.

A detailed list of bugfixes and features will be released after successful firmware deployment. Stay tuned!

Firmware 2.0 Launch

We are currently preparing the release of our Firmware 2.0, which integrates lots of new check-up and monitoring scripts to mitigate all issues that several people had in the last few days, like the “Wrong Wallet Adress” error, Dashboards that shut themselves down, and more.

If you already wrote us an email in the last few days and didn’t got a reply already, you will get a reply today and will get 2.0 first. Currently, roughly 50 devices have received the update and we are monitoring the quality at the moment. Things look very promising – all 50 devices run exactly as planned, and if errors happen that are out of our control, the device tries to fix it by himself (in 2 cases, something happend, and the miners automatically healed themselves within seconds, causing no disruption in service).

More infos will be published over the weekend!

Miner Stability Improvements

The larger our fleet gets, the more errors we are able to identify that cause instability to Helium’s miner software – this makes our miners more optimized and reliable over time.

Some users were still experiencing sync problems or hanging miner software due to a misconfigured configuration file that should be there to increase sync speed. The issues were identified over the weekend and every miner is currently getting a fix update. Please give each miner roughly 6-8 hours to show you some status in Helium Explorer or Hotspotty Explorer and ideally don’t disconnect it over that period.

Here’s a technical explaination on what happened:

There is a configuration file called docker.config that can override several settings in the sys.config file of the miner docker container. This configuration file gets altered mainly by the instasync script and on first device flash, as well as some other scripts sometimes.

If this configuration file gets corrupted, the miner software sometimes (not always, but sometimes) has problems to correctly boot up. We have done the following now:

1. We replaced the configuration file with a working one and blocked scripts to alter the file. That’s why instasync is currently disabled (and will be re-enabled in the next few days).
2. We are grabbing a snapshot from the blockchain that is at max 240 blocks behind and adding it to the miner data folder.
3. We load the snapshot with a miner docker command and resume the syncing.
4. We double-check if syncing is resumed with another command.

If you want more detailed information, please let us know, we’re glad to be as transparent as possible here.

We want to especially thank our early customers for the early trust you are giving to us. We understand that things in the Helium space can sometimes be frustrating, and you can imagine how we as developers feel when something doesn’t work out as well as planned even after rigorous testing. Sometimes, something just shows up if the fleet gets larger that wasn’t caught in the testing before. That’s typical life as a software developer.

Thank you all! More updates about dashboard functionalities and fixes coming soon!

Dashboard 1.02 Release Announcement

We released an update to our Dashboard today, your miners should now be on Version 1.02

Fixes

• For users who got the dashboard via OTA update, it can sometimes happen that the dashboard shuts itself off after a while. The cause for it was found and looks fixed in v1.02. We will further monitor if shutdowns still happen and release another update next week if we see something unusual.
• Instasync was removed in Version 1.02 as it was causing problems for some users, we will investigate the issue and re-enable it again in V1.03.

Features

• New button: Miner Height
• New button: Miner Name
• New button: Miner P2P Status

As the info summary command takes lots of time (sometimes over a minute) to gather all the info, we wanted to add new separate checker buttons that are way faster, if you just want to check your miner height, relayed status or name.