Edtech Journal 2015 V1 Engineering Review
CETPA Engineering Review captured using Gambassa LMS

Engineering Review: Content Filter

By Phil Scrivano

Abstract:

The study features four vendors, each with their own methodology to solving the content filter problem. This engineering review focuses on the user experience of students, teachers, and administrators in a real time simulation of a classroom environment. Based upon data consisting of 2,000 observations, it is clear that each methodology offers strengths and weaknesses that must be taken into account to determine the overall sentiment and effectiveness of any one content filter. Our conclusion demonstrates that there is a wide array of choices available for districts that must be evaluated according to the local needs and culture.

An Engineering Review Versus Shoot Out

An Engineering Review is where vendors can bring their products into a live setting and get real time feedback from students, teachers, and IT staff. Data is collected and measured in both rubric format and freeform comments, as well as 3 Likes and 3 Dislikes. The live events may consist of one or more vendors on the same day serving as reference points of comparison. The rubric is designed to not favor any particular vendor or product design but rather the needs of the customers. CETPA’s role in providing this opportunity is two-fold, to provide a conduit for districts to review products, whether prototyped or officially released, in a live setting and to provide vendors the ability to measure the user sentiment, user behaviors, or product efficacy in real time. The Engineering Review also allows the Vendor a freeform discussion format to either clarify, address, or seek additional feedback from the testers. This dialog can be very powerful and useful for companies testing new features, designs, and products before they are released or getting live feedback on products that are already deployed. The purpose of an Engineering Review is for Districts to simulate real time usage of products and to evaluate potential products. It is not the intent to punish any vendor or product for performing poorly, especially unreleased products. We feel that creating this dialog between district and vendor creates a compressed feedback loop. Vendors may select their own rubric or use one created by CETPA. We prefer they’d use the CETPA developed rubric in order to allow for the greatest calibration of results, however, this is not mandatory. 

The goal of an Engineering Review is to Test, Learn, and Compare. Results will generally characterize the pros and cons of a product design concept and feature concepts, and may not specifically address the product itself. The Reviews may also forecast new trends or highlight upcoming features to be found in the marketplace.

Why an Engineering Review is NOT a Shoot Out

A Shoot Out is an opportunity for officially released and Vendor sanctioned products to go head-to-head with other officially released products on an agreed CETPA rubric. The rules of a Shoot Out are different than an Engineering Review in this subtle way: The rubric’s results will go through CETPA membership peer review and be made public in the EdTech Journal. The vendors will be allowed an option to comment on their feedback results after the engineering and peer review.

The goal of our EdTech Journal Reviews is to give our membership information based on our own professional evaluation, teacher response, and student use of education products. With the premise that no technology is perfect and every district will have features that are important to them. All four vendors who stepped up to this form of evaluation deserve our respect, support and a big thank you. Although each vendor contributed financially to cover this journal's production cost, all content is free of vendor influence and CETPA remains unbiased.

The study features four vendors, each with their own methodology to solving the content filter problem. This engineering review focuses on the user experience of students, teachers, and administrators in a real time simulation of a classroom environment. Based upon data consisting of 2,000 observations, it is clear that each methodology offers strengths and weaknesses that must be taken into account to determine the overall sentiment and effectiveness of any one content filter. Our conclusion demonstrates that there is a wide array of choices available for districts that must be evaluated according to the local needs and culture.

Introduction to content filters study

Education networks have become the largest controlled networks in the world. Keeping the environment safe consists of three lines of defense, the firewall, the content filter, and staff. In a school district the firewall is the gatekeeper of all traffic in and out of the network. The content filter categorizes all content and applies rules according to acceptable use policies to determine if the content should be allowed. This engineering review gave us the opportunity to experience a live simulated classroom while observing the impact and performance of the content filters on learning activities.

Our current CETPA Content Filter engineering review featured ContentKeeper, iboss, Lightspeed, and Securly. The live event was held on January 9th, 2015 and hosted by the Ventura County Office of Education. Our reviewing team consisted of school district technology professionals representing 49,287 students (Sorce: www.ed-data.org 2013-14 ADA), three certificated teachers representing elementary and secondary, and 14 students representing grades 5 through 11.

Titles Represented:

  • Chief Technology Officer
  • Database/Network Analyst
  • Director Of Technology
  • Lead Technology Specialist
  • Network Specialist
  • Network Systems Analyst
  • Student
  • Teacher
  • Technical Supervisor
  • Technology Specialist

Districts:

  • Filmore USD
  • Montecito Unified School District
  • Oak Park Unified School District
  • Ocean View School District
  • Oxnard Unified
  • Placer County Office of Education
  • Ventura County Office of Education
  • Ventura Unified School District

The review process started with building a vendor-approved rubric. The rubric produced over 2,000 data points including numerical data and reviewer feedback. The day was organized into two parts; hands-on engineering review in the morning and Q&A session with a sales person in the afternoon. During the morning engineering session each vendor placed their product in-line with a room specific wireless access point on the inside LAN port and unfiltered WAN connection on the external port of the filtering device. The exception to this setup was Securly which is a web hosted solution that required routing changes to the network during the Securly demonstration. CETPA professionals and teachers were given direct access to the management web interface of each product in order to score rubric questions and interact with the vendor engineer. All other vendor employees were out of the room during the morning session. Students were given the task to evaluate if the product was over-blocking or under-blocking education content based on researching assigned topics: gun control, human organ trafficking, and legalizing marijuana. Students were also give the additional task of evaluating if they could circumvent the content filter and if so, how hard was it. A note needs to be made that student selection was based on best and brightest at grade levels and interest in technology.

Rubric Details:

CETPA professionals and teachers shared the same rubric while students used a separate rubric based on the end user. Two example adult rubric questions:

Rate a 1-4 score for the description that most closely matches your evaluation

Admin Interface Rating
1 => Design is confusing and very difficult to navigate
2 => Design lacks intuition and could be easier to navigate
3 => Design is clear, but assistance is needed to navigate some areas
4 => Design is clear, easy to navigate, logical, and provides an effective means to administer the filter
Comments:
Local Allow and Local Block Rating
1 => Site requests for review are difficult to manage
2 => Site requests for review are time intensive to create rules for
3 => Site requests for review are easy to manage and create local allow/block rules
4 => Site requests for review are easy to manage and create local allow/block rules based on groups or other criteria
Comments:

During the morning engineering session each vendor was able to watch scoring and comments in real time. This empowered our vendors to prepare and respond during the afternoon session to reviewers' concerns and feedback. Vendors could see the information, but the contributor's identification was kept private. This setup made for a focused and direct discussion in the afternoon sessions. The following evaluations are based on scoring, comments, and a like/dislike response from each evaluator.

Findings

The order of vendors below reflects the reviewers' overall scoring based on 2000 data points. The content within the vendors product write-up reflects the thematic observations and sentiments of the group.


 



iboss Cybersecurity

Likes:

The top feature listed by all three groups, IT professions, teachers, and students, is the iboss content filter's ability to view web content that is encrypted between two hosts such as Facebook and a user logged into Facebook. SSL Encryption was originally developed by Netscape to secure MasterCard transactions over the Internet. Today, Facebook, Google, and many other web-based social media use this technology to protect the privacy of adult users. In a public school setting there is no expectation of privacy and sadly, being able to alert of social behaviors such as bullying, suicide, and harassment is part of maintaining a safe environment for learning. iboss accomplishes this without the need to install an agent on devices on the network.

Related to student safety is the ability to force all YouTube videos through a hosted search type engine called Cleanvideosearch.com. This engine strips user-feedback and advertisements plus inappropriate video content. The importance of this level of safety was expressed by a third grade male student's mother who was forced to discuss sexuality with her son due to inappropriate pictures displayed to the student's computer. The parent had hoped to approach this subject much later. There is no undoing the impact of this content on a child's mind.

Teachers were impressed by the override abilities of this filter because a teacher is able to view what the student rule-set will allow or block from the teacher workstation; no trial-and-error while looking over the shoulder of the student.

iboss technology goes beyond basic filtering by most notably applying threat rules that monitor much more than web content, such as identifying malicious behaviors traversing network (backdoor communications, botnets, DDOS attacks, chat rooms etc.) As stated by one IT Professional, “very impressive and clearly the most robust from a filtering and traffic shaping point of view.”

Bandwidth shaping was the second theme that users liked. iboss has the ability to limit how much bandwidth a user or group can use while at the same time guaranteeing maximum bandwidth for certain destinations such as CAASPP testing.

Students at all levels reported that iboss did the best job of not under-blocking or over-blocking content while being one of the most difficult to circumvent.

All three groups found the graphical interface controls well laid out, easy to navigate, and flexible.

Dislikes:

The iboss product relies on Active Directory / LDAP groups to apply policies. With no classroom level grouping ability, the teacher is unable to create instantaneous policy changes for overrides. This possibly limits teacher ability based on directory services groups and makes management of the filter dependent on a well-staffed and approachable IT department.

Licensing is based on device counts versus ADA, Average Daily Attendance, which is verifiable at http://www.ed-data.k12.ca.us. No matter how the price is calculated, the chief business official produces a cost model based on price per student. It is less complicated when companies give straight forward pricing based on cost per ADA up front.

One secondary student articulated an important viewpoint that needs to be part of any technology security product. He writes iboss is “Extremely un-private,” staff “know exactly who, what, when, where, and such when accessing anything. Teachers could use it to take students' private info, even though it's unlikely, you never know.“

Vendor Response:

Although pricing typically based by device which is common with web security products, iboss can and has offered pricing based on ADA should it be desired.

With regards to the secondary students comment, we would like to clarify that this is a strength of the iboss product. iboss can provide detailed, user based reporting of all data that is not sensitive (i.e. Financial). This level of visibility is unmatched and while any solution with this level of detailed reporting can be used for purposes outside of its intent by administrators, it is unlikely, because those managing the systems are professionals. It’s important to note that this level of detailed reporting is essential for investigations during incident response. This greatly reduces the burden on resources and time during investigations, while also providing clear visibility into what actually occurred.

That said, the ability to tune the reporter to provide less visibility is possible if desired.

Robert Erwin - Director, Marketing




Peer Review Comment:

John Patten 05/20/2015

Nice review! As we are current iBoss users, I would also like to share that the reporting, such as threat reporting can be very useful. In a pinch we have used the realtime threat reporting feature to track down malware infested computers. Also, like many county offices of education, Stanislaus County Office of Ed provides the iBoss filtering service to districts that select that service. The iBoss management capabilities allows for individual groups to be created and managed independently by the subscribing district. This gives the member districts a quasi-cloud based solution as the member district receiving the service does not have to worry about managing the hardware. This makes iBoss a nice solution for small to medium sized districts in a shared, but independently managed, environment.


 



Lightspeed

Likes:

Lightspeed reporting was rated “best in class” by all three groups participating, IT professionals, teachers, and students. There is a robust menu of standard reports from web activity to suspicious web searches. Each report has an additional customization panel to the right which enables the administrator to drill down on specific users, IP addresses, time, protocols, etc. A top performer is the Suspicious Search report which has a filter option to summarize the report on several criterion. Using the summarize option by user produces a pie chart graphic at the top of the report that tells the story for non-technical people. As robust as the out-of-the-box reports are, Lightspeed offers custom reports to meet the individual needs of a district. This is an important function because Lightspeed recognizes that there is no such thing as a typical school district and each district has individual cultures and needs.

Another strong like for this product is the teacher override function which enables teachers to override filter settings to a certain level. It is important to recognize that some categories such as “child pornography” should never be allowed to override in a school setting. Typically what happens is a user makes a typing error that has the potential to go to an inappropriate site. Overrides can be device and group based depending how the directory services are set up and the type of devices being used. Students report it has “WebZones” for easy and quick content unblocking for a class.

Other likes for this product were the accuracy of the database update process that includes data from all school districts' local allow and local block activity. Help documentation is user community based and the company provides oversight of the information for accuracy.

Dislikes:

Students reported that Lightspeed does not block new proxy type servers not yet in the database and is vulnerable. The product should detect a proxy type server is being used and block it in real time. Another strong student concern was under-blocking on most Google images. Students also reported that most of the blocked websites can be opened by putting “https://” in front of the URL. The “s” in “https://” creates an SSL, Secure Socket Layer, encrypted session for the site. Another student concern was blocking of important educational sites because all “Blog” type sites are blocked. This is unacceptable and should not be labeled bad just because sites have open feedback capability. Ability to access sites through VPN programs such as “Hotspot Shield” is also a concern.

IT Professionals reported that Web Activity and Suspicious Search reports are the most used reports, the basis of all user searching. But these reports are buried towards the bottom of the web page.

Common themes were the lack of in-line version-proxy blocking, no bandwidth controls, does not seem to inspect SSL, and site encryption is strictly block/unblock which causes issues with YouTube and Facebook.

Vendor Response:

Many issues cited (including lack of SSL inspection, lack of granularity in handling https sites, and unblocked proxy sites) were the result of an equipment issue during the review. The on-site Rocket appliance was not able to be installed and therefore filtering policies were not able to be adjusted for the evaluation.

Rob Chambers - VP Product Development




ContentKeeper

Likes:

ContentKeeper distinguishes itself with the ability to inspect and manage SSL and Web 2.0 content which allows a more finely tuned experience with social media sites. For example, it was demonstrated that Facebook can be allowed while at the same time blocking inappropriate comments within a posting. Top level domains are sites such as Facebook, Craigslist, and YouTube. Within a site such as Craigslist, the section titled “Personals” can be blocked while allowing “Jobs” to remain open. This is done by deploying a trusted root certificate to each client enabling full decryption and inspection of HTTPS content on the district network. Google is leading the charge to have most web content delivered in an SSL encryption format. ContentKeeper's SSL full decryption can also set limits on particular pages and selectively decode traffic. Sites such as Bank of America can be set to fully retain user privacy to prevent ID theft.

Another significant benefit of a certificate based connection between the client and the filter is near wireline speed over the network and through the content filter. This is important as districts are choosing hosted solutions such as Google products and learning management systems and have Internet connections exceeding 1GB speeds.

The ContentKeeper database runs differently than the other companies reviewed. ContentKeeper does not go out to scan websites, but relies on user requests to identify sites/pages to classify. The database has 92 categories and works on the premise of learning and healing based on user experience on the Internet. There are several levels to this technology. The first question is how a new site that is unknown to the database is handled. There are two choices local administration can make in this situation based on local culture and direction of the superintendent. Choice one is to block new sites unknown to the database. ContentKeeper states that the site will be categorized within 24 hours.  If a local administrator adds the unknown site to the local database, the site will be re-categorized instantly to what ever category the local administrator selects. The second choice is to allow unknown sites. The site will be evaluated and categorized within 24 hours. As with choice number one, administrator intervention accelerates the categorization process. The database “heals” by receiving notification of any categorize changes made by local administrators. Sites are not automatically recategorized, but follow a process similar to unknown sites. All three groups of reviewers commented that this product has good policy and role selection across grade levels and site groups that authenticate via Microsoft Active Directory, proxy authentication, radius authentication, and teacher coaching that enables policy elevation for normally blocked sites.

Student feedback for ContentKeeper was the most positive. Students had a difficult time circumventing the filter. Only one student found a way around it which was outstanding considering the caliber and grade levels of students in the room. Students liked the graphical interface and stated that there was almost no frustration over sites that were blocked. Students were impressed with the ability to block part of website, ie. Facebook posts. One high school student stated noted that “you could go to lots of websites that were school related that are usually blocked by other filters.” These sites were unblocked for the desired content while at the same time inappropriate pop-ups or content within the site, at the page level, was blocked.

Dislikes:

There were two common themes for the dislikes of the ContentKeeper product. The first was concern over the administrator user interface. IT professionals felt it was not intuitive, hard to find appropriate settings, and that there is a lack of “Breadcrumbs” or navigation links that help you to see where you are in the hierarchy/tree of a sitemap. Some felt the interface had a “90's” look and feel or at a minimum is outdated.

The second feedback was about requiring a default email client on the user machine if the teacher or student needs to request a site be opened or blocked. Many districts have gone to web based email systems which do not require a client setup. More than one company evaluated had the same issue. One company did have a web based direct email communication setup like most web based stores. Look for this to become standard on all products.

Vendor Response:

ContentKeeper has some great plans to simplify the interface, the first items of which can be seen in our new Gen II Report Central dashboard components (V166.47 and newer). It provides a very simply to use, one click reporting overview with very intuitive report field customization. I have attached an example - it’s a one click customizable dashboard block report. We also have a new CK-Mobile Admin App for the iPhone which allows ContentKeeper administrators to control the box for anywhere. This App was developed very quickly and is driven totally via our new interface API.

ContentKeeper has been working on separating the user Interface layer from the underlying filtering and configuration code for over two years now. The driver for this initially came from a large Telco filtering project we are doing with Chunghwa Telecom (in Taiwan, they are the major telco there) who needed an interface in Chinese. That project produced a secure XML based API within ContentKeeper. This API separates the interface totally. This allows us to easily render any new interface we like in any language quickly and efficiently. We are expecting this to provide us with a major competitive advantage in the years to come as demands for simplicity and different styles of interface design evolve and change. The XML API reduces the development time to render a new interface from years of programming to just a few weeks of programmer time.

Breadcrumbs can easily be added so that will go into the mix also.

You should see ContentKeeper's user interface evolve quickly now to address the demands we are seeing for intuitive, simple to use, uncluttered interface design. So keep watching our releases over the new 6-12 months and I am sure you will be suitably impressed.

David Wigley CEO ContentKeeper Technologies



Securly

Likes:

Securly was the new player on the block for educational content filtering. The product is easy to implement, can filter encrypted traffic, and is fully cloud based, requiring no internal hardware. Firewall settings are configured to require traffic to go through the hosted filter. This means this is currently a low ongoing cost solution based on no hardware to maintain and setup.

This company seems to have a new and interesting take on Web filtering such as detecting and identifying bullying and self harm. Social media reporting for encrypted sites is done for Facebook, Twitter, and Google Plus postings; although no other social media sites at this time. Securly for parents shares what students are doing online and tracts trends.

The group's consensus was that it had an easy to use user-interface that can block or unblock sites quickly and effective login and submit feature for submitting requests to the company. Students reported that the filter was reasonable in what it blocked and allowed.

Dislikes:

Still new and being developed. The administrator interface is basic, but does not provide much depth of use. It is a product that looks like it's in its infancy as in beta.

Another major concern is that users can only authenticate via Google based authentication, however, Microsoft Active Directory authentication is promised soon.

Teacher overrides apply to entire school site so this would not apply to just their classroom. A teacher, utilizing a student computer, logs in to a teacher's Google account in order to set up white-list entry. Once the teacher logs off, the entire site is white-listed for the whole school. There is no way to white-list sub domains. A teacher can temporarily override for the entire site, but IT needs to make final approval.

Block request launches default email client which is surprising since Google integration is a key element. There is currently no customization for the block page users get. Opening and blocking sites is currently a manual process that is associated with the appropriate group such as students.

The database updates process is currently hidden because it is cloud based. Content categorization is done by a third party located in Israel (Komodia). There are currently a limited number of categories and it was unclear to our reviewers how much focus is on K-12. Categorizations apply to entire domain which means there is no way to whitelist sub domains. The filter always allows unknown content through and then reviews. Database needs improvement and needs finer control of policy exceptions.

At this time there is little in the way of reporting other than searching through at the audit trail tool to generate report of student usage. Keeping an eye on the dashboard "latest Activity" and audit trail would be necessary to drill down into issues. Basically, there is no reporting feature at this time.

Categorization rules affect an entire domain as such, there is no way to whitelist specific sub domains. The rules logic for categorization needs to have more granularity. The lack of specificity in filter rules left the impression that the product was not fully developed.

Students reported that there are lots of key words blocked, such as marijuana. After marijuana would not show the student reported trying to search for “Diet Coke” and “Chicken Breast Recipes,” and neither worked. One student got one blocked site and seven unblocked sites that should have been blocked. Due to being term-based filtering, If you typed the keyword halfway or partially, you could access what was blocked and there are only a few sites that were blocked. For example, Securly blocks keywords on search engines, but you can still go onto sites like Gambling.com, Minecraft.net, and Steam. Students also reported that Securly was the easiest filter to bypass using unblocked tools such as Tor, and VPN Hotspot Shield.

Vendor Response:

Since the evaluation, we have significantly revamped our approach to filtering. We have phased out Komodia for filtering and rely on our PageScan technology to add dynamically to our filter database. PageScan works as follows: (i) Each time any of our 500K users world wide go to a site that is not classified in our database, we use offline analysis (scanning HTTP response content) to dynamically classify the site as belonging to one of several CIPA categories. (ii) Any subsequent access to that domain across any of our customers would then be blocked. PageScan is currently live in production and is scanning 100K new sites every week for inappropriate content that is then pro-actively blocked.

We have used User Experience studies (and continue to do so) with the target group (IT admins) to come up with a User Interface that is clean and easy to navigate. Several of our customers love this aspect of our product. Perhaps one way to go as we continue to add features is to have a "Power User" section.

We are still considering AD integration and will very likely execute on it at some point. The question really is not so much "Can we do it?" but "Should we do it?". With Chromebooks taking off virally in the K-12 space, most schools we speak to (including the larger districts) seem happy with Google SSO. So implementing this additional functionality would simply be a function of market demand which we are not seeing enough of yet.

The ability to whitelist sub-domains has been introduced since the evaluation.

The Teacher Whitelist was meant to give teachers over-ride privileges without Admin intervention and that is exactly what this feature does. In fact, since the evaluation, we have extended this functionality to allow teachers to whitelist YouTube channels that are not found in YouTube for Education.

User reports (ability to pull a PDF report on a single user in a given date range) has been introduced since the evaluation.

Bharath Madhusudan - Co-Founder/CTO



Conclusion

The study features four vendors, each with their own methodology to solving the content filter problem. This study highlights the user experience implications for each methodology from the perspective of the student, teacher, IT and administration. What was not covered here, and should be included with any discussion concerning comparative technologies, are user privacy, total costs, and the efficiency of getting users on the network.

Phil Scrivano CETPA
Special Thanks to the Ventura County Office of Education for hosting the event.
Authored by Phil Scrivano