12 Replies - 481 Views - Last Post: 29 November 2018 - 12:38 PM

#1 dreadstarx   User is offline

  • New D.I.C Head

Reputation: -2
  • View blog
  • Posts: 30
  • Joined: 09-November 18

Web Project Interacting with Local Environment

Posted 27 November 2018 - 10:37 AM

Hey Guys/Gals,


Since my work killed my idea for a project, suggested I come up with a project. Well, I've got one but again, a bit out of my scope of knowledge. I've done web programming before, nothing fancy or super glorious but still a major accomplishment for myself.

Here's the project and where I'd like advice. It didn't seem to fit any specific language for help, so I thought this would be the best place to ask.

One of my hobbies is archiving. I archive everything, my Google Drive account is well over 400TB of content, and I add about 800GB of content a day. Not bragging, nor do I share. This is just a personal hobby that's shared by others from /r/DataHoarder. We enjoy archiving the internet, mostly sites that are dying and decide to shut down. We write scripts to scrape the content, and store it. Eventually we hand it off to InternetArchive.org. Anyways, to the point.

I'm sure some of you have heard of ZFS, it's a type of software RAID. One of the few that doesn't have a GUI. Everyone says it's SUPER easy to setup, and maintain. That's a lie. Complete and utter lie. Out of everyone on DataHoarder, I probably have the largest ZFS Arrays. I'm sitting around 500TB of local usable capacity, in total I'm sitting on about 1.1PB of capacity. I just don't have the other half in use because of budget constraints, and people not keeping their word.

I'm looking to write a nice, fancy UI with loads of bells and whistles. I started talking with a friend whose a Web Programmer, PHP/SQL, etc. He told me it was frowned upon to access the local files via web scripts. It's a security risk, which got me thinking.

What languages would one use to accomplish this? I'm not the only one whose after this, there are a LOT of people who don't know how to work with ZFS, and learning it is a struggle. I had to wipe my arrays and rebuild them multiple times to get it right. And learning regex on the fly to gather all of my disk-by-uids, not fun nor easy.

Here are some of the things it'll need to do.

Install ZFS + Necessary Tools
Configure ZFS
- 3 different ways of making the arrays, by drive letter (sda/sdb/sdc), disk-by-label or disk-by-uuid.
- Drive Management / smartinfo via smartctl
- Error Logging
- Permissions
- Resilvering (rebuilding the array from drive failure or drive removal)
Notifications via Email/Text
Graphs via Grafana
Statistics saved to InfluxDB (Friend works for the company, so I have loads of support there)

I planned on using these applications with it;

zfs
InfluxDB
Grafana
smartctl
nginx
Let's Encrypt

This is something I'd like to work on completing or at least getting a beta test done by spring of 2019. Give me your thoughts!

Is This A Good Question/Topic? 0
  • +

Replies To: Web Project Interacting with Local Environment

#2 Skydiver   User is online

  • Code herder
  • member icon

Reputation: 6656
  • View blog
  • Posts: 22,740
  • Joined: 05-May 12

Re: Web Project Interacting with Local Environment

Posted 27 November 2018 - 01:54 PM

Perhaps I'm missing why you need a web interface to interact with the local environment. Is it just a cheap way to get a GUI that will work for different platforms without going down the Qt, or Tk path? Nowadays, it seems like Electron is coming out to be a popular choice to target multiple platforms but also get a good looking GUI.
Was This Post Helpful? 0
  • +
  • -

#3 dreadstarx   User is offline

  • New D.I.C Head

Reputation: -2
  • View blog
  • Posts: 30
  • Joined: 09-November 18

Re: Web Project Interacting with Local Environment

Posted 27 November 2018 - 04:08 PM

ZFS is very unforgiving. A lot of people prefer GUIs rather than looking at CLI all day long. Not to mention, you need to have multiple windows open to gather all the information together. In terms of skill for Linux, I'm still a beginner but I'm about to finish my Linux+ Cert.

The reason I'm doing this is because there isn't one but I've seen multiple ask for it. Everyones solution was to just write a script and have a php script parse it all into Human Readable Text. That's all well and good, but no one likes to stare at text documents all day either. The other reason is, there's no single way to manage multiple ZFS Arrays without having TONS of windows open. With the 5 Arrays I have, and the 100+ drives I've got, having a universal way to manage them would be beneficial to the community. I know someone whose got over a dozen ZFS arrays, and he uses scripts / ssh keys to gather all his information. Wouldn't it be easier if a single server had something running to collect all the information and display it?

What i'm looking for is the best way to approach this, language wise.

Should I write an application and have a website talk to it in order for commands to be ran? Do I write it all in a web language and have it issue commands that way?
Was This Post Helpful? 0
  • +
  • -

#4 Skydiver   User is online

  • Code herder
  • member icon

Reputation: 6656
  • View blog
  • Posts: 22,740
  • Joined: 05-May 12

Re: Web Project Interacting with Local Environment

Posted 27 November 2018 - 05:43 PM

I wasn't questioning the need for a GUI (unlike those other people on Reddit who claim that managing ZFS is easy). I am questioning the need for a web GUI as opposed to some kind of native GUI.
Was This Post Helpful? 0
  • +
  • -

#5 dreadstarx   User is offline

  • New D.I.C Head

Reputation: -2
  • View blog
  • Posts: 30
  • Joined: 09-November 18

Re: Web Project Interacting with Local Environment

Posted 28 November 2018 - 10:51 AM

Ahhh... My apologies, I read that wrong. I find that a Web GUI is easier to work with than a native gui. I was also looking at remote management, being able to host it on a single server, and have it interact with it. I've come up with a few ways to handle this, after your question I started to think about the ways to do this.

1. WebUI that interacts with a application running on each server.
2. WebUI that has multiple modes. i.e. Server / Client / Client + Server
3. Native GUI with an application to monitor them all remotely instead of logging into each one individually.


These were what I came up with. I'm trying to work it in to where the user has a lot of control, and add features that I know I'd like. My in-depth knowledge of the Linux System is pretty weak, but while I figure out a layout and game plan, I'm learning about it.

I was looking at PHP7 for the web language, and the applications side would be C++. However, you guys know better so let me know what the top languages would be. Please, for the love of all sane things, don't say node.js. I might die inside, haha. I have an acquaintance that lives and breathes node.js, says it's going to replace all other languages. Same guy also says that Rust will replace C++, and C++ isn't worth learning anymore. Anyways, I'm going to think on this more.
Was This Post Helpful? 0
  • +
  • -

#6 BetaWar   User is offline

  • #include "soul.h"
  • member icon

Reputation: 1581
  • View blog
  • Posts: 8,401
  • Joined: 07-September 06

Re: Web Project Interacting with Local Environment

Posted 28 November 2018 - 12:50 PM

I, personally, don't see a web GUI as a bad thing for native applications. In fact, if you look around, a number of applications out there run web servers locally that interact with your local files and present things in a web browser (which they typically package with their server). For instance, steam is in the process of moving over, discord already uses Electron, a lot of enterprise applications give a web UI to configure your system (be them software or servers). Heck, even I prefer to make my applications written in C++ to interact with the system and be fast, but present all the UI in a web browser as it gives me very quick turn-around time on UI modifications and development while also giving me all the power I want for the back-end.

One thing you may find of interest is FreeNAs, which sounds like what you may be looking for. It uses zfs behind the scenes and then lets you allocate and manage all of your storage through a web UI. I have a system at home set up to use it for backing up home machines (though the system is only 80TB, instead of multiple PB).
Was This Post Helpful? 0
  • +
  • -

#7 dreadstarx   User is offline

  • New D.I.C Head

Reputation: -2
  • View blog
  • Posts: 30
  • Joined: 09-November 18

Re: Web Project Interacting with Local Environment

Posted 28 November 2018 - 01:29 PM

BetaWar I'm well aware of FreeNAS, however their development team has some issues to work out. It uses ZFS but I use Ubuntu Server as my base OS, and I have other applications not supported by FreeNAS. I've been told by members of /r/DataHoarder to steer clear of it. Since you use C++ to interact with the backend, I'll use that as my primary backend language. My question is; what do you use for your web languages? PHP? Node.JS?

The goal is to make this work on all Linux flavors/distros Without too much meddling. It needs to be about 75% plug n play, but that 25% is the configuration options. Once I have the 2 languages, I can start planning, I've already got a decent UI. I'll just need to install ZFS on my laptop and refresh my memory of the commands.
Was This Post Helpful? 0
  • +
  • -

#8 BetaWar   User is offline

  • #include "soul.h"
  • member icon

Reputation: 1581
  • View blog
  • Posts: 8,401
  • Joined: 07-September 06

Re: Web Project Interacting with Local Environment

Posted 28 November 2018 - 01:43 PM

Depending on what you are wanting to do, you may actually want to look at Go for your back-end and web language. It is capable of very quickly setting up a web server and fairly easy to add handlers for web pages. I don't know exactly how well it interacts with zfs, but there may be a library out there for it to make things easier there.

Normally, I try to use the same language for my back-end server interactions and my back-end web development; so most of the time that is C++. However, Java, python, Go, and a number of other languages will make your life a bit easier (especially if you already know them). In the case of C++, I have actually built my own web server in C++ to deal with everything and trivialize API creation (as well as static web page serving); so that is what I use.

For front-end languages, I stick with the simple stuff: HTML5, CSS3, and vanilla JS.
Was This Post Helpful? 0
  • +
  • -

#9 Skydiver   User is online

  • Code herder
  • member icon

Reputation: 6656
  • View blog
  • Posts: 22,740
  • Joined: 05-May 12

Re: Web Project Interacting with Local Environment

Posted 28 November 2018 - 03:15 PM

My major concern with a local web server is that typically when somebody puts together a local web server the assumption is exactly that: "It's for local use only. I don't need to worry about security. It lives behind my internet facing firewall, and I trust everybody on my network." That was exactly the same mentality that went into hole taken advantage by the Slammer worm: SQL server is supposed to be running on a LAN and firewalled off from the rest of the Internet. (Granted the other thing that made Slammer particularly bad is that SQL server was being installed and left running by everybody and their brother due to Microsoft's (successful) campaign to get people to use MDAC.) The question became: "Do you really need a SQL server running on your desktop?" And so I pose a similar question: "Do you really need a web server running on your desktop?"
Was This Post Helpful? 0
  • +
  • -

#10 dreadstarx   User is offline

  • New D.I.C Head

Reputation: -2
  • View blog
  • Posts: 30
  • Joined: 09-November 18

Re: Web Project Interacting with Local Environment

Posted 28 November 2018 - 03:31 PM

I'll look into Go. The one thing I know about Go is that it's pretty much universal, and has no dependency issues. I was hoping to stick with C++ / PHP/CSS3/HTML5 for everything because I'm more familiar with it, but I guess it might be time to learn an actual language completely.
Was This Post Helpful? 0
  • +
  • -

#11 Skydiver   User is online

  • Code herder
  • member icon

Reputation: 6656
  • View blog
  • Posts: 22,740
  • Joined: 05-May 12

Re: Web Project Interacting with Local Environment

Posted 28 November 2018 - 09:03 PM

Although above,mI come across as being opposed to running a webserver to make a local device, I do realize that any typical tech savvy home would already likely have a router, a printer, and/or a NAS all running web servers on their own to provide management GUIs. So what the heck, why not add another web server to manage your ZFS drive arrays.
Was This Post Helpful? 0
  • +
  • -

#12 BetaWar   User is offline

  • #include "soul.h"
  • member icon

Reputation: 1581
  • View blog
  • Posts: 8,401
  • Joined: 07-September 06

Re: Web Project Interacting with Local Environment

Posted 28 November 2018 - 10:24 PM

:) I work in the storage industry and can tell you that at least 2 major players in the storage game have web-based management UIs for everything from users, to fault detection, and volume creation.

Also, if you want to stick with C++, look at nginx, it is a C++ web server that should allow you to write C++ code to add web handlers to it, and it is open source. We use it at work a bit, along with pion (though that is far from a "good" solution, so I don't recommend it).

Go's standard library has a http server as part of it, which is nice.

Alternatively, you could always go with Apache and have it run PHP scripts and then have those scripts call other applications as necessary to do the ZFS management. That would work perfectly fine, there isn't any reason you couldn't approach the problem that way.

Sadly, this is one of those areas where there really isn't a "wrong" way to do it, but there are a lot of potentially correct ways. I would argue that if you tried to do a HTTP server in bash you are doing it wrong though...
Was This Post Helpful? 1
  • +
  • -

#13 dreadstarx   User is offline

  • New D.I.C Head

Reputation: -2
  • View blog
  • Posts: 30
  • Joined: 09-November 18

Re: Web Project Interacting with Local Environment

Posted 29 November 2018 - 12:38 PM

Skydiver Very true but a lot of people I've come across, don't use builtin web servers. They actually dislike them. I couldn't care either way, as long as it offers the same amount of control that a separate server would offer. But I'm looking at this from two points, 1. Beginners and 2. Advanced. I'll think on this more.

BetaWar I'm quite familiar with EMC and NetApp. I work with both of them, I love EMC's UI, but hate how bloated it felt. Not to mention, half the people who configured the arrays were numpties, and couldn't do it right. I learned to ask for help when working on either of them, a fellow colleague is an SME when it comes to NetApp/EMC. Even my EMC friend whose sole job is to setup, configure, repair, and troubleshoot, comes to him for advice.

I use Nginx only, I gave up on Apache years ago. It's good but I felt that Nginx was easier to work with and understand. My biggest fear is a security problem. Hey now, I was going to use Python's web server ;)
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1